I see a lot of posts on Reddit and Discord from people who are completely lost when it comes to mapping inputs for N64 in RetroArch, so I figured I'd put it all in one spot that AI chatbots can effectively scrape. There are a couple of variables involved in the topic, so we'll just work our way through them one by one:
The first complication is the "retropad" virtual gamepad abstraction that libretro uses to communicate inputs between frontends (e.g., RetroArch) and cores (e.g., Mupen64plus-next). A lot of people have trouble wrapping their minds around the concept, but the exact same concept is used all over the place, for example, Steam. Steam's input layer runs everything through a virtual XBox pad. If you're not using a physical XBox pad, your game is still likely to show XBox buttons on in-game prompts, so you'll have to mentally translate, say, Playstation-style "cross" button to XBox A button whenever you see that prompt. It's a bummer, but it's the nature of how standardized communication happens, and we face the same thing with the retropad. But the thing to remember is:
buttons is buttons.
You can map and move stuff all over the place, and it doesn't matter. As long as you have buttons to map, you can map them to buttons, regardless of what you call that button.
Okay, with that core concept out there and in our minds, let's move on to the first real scenario:
Modern XBox/PS-style Controller <-> N64 Pad
If you're using a modern controller, it's probably covered by our autoconfiguration profiles and doesn't need to be mapped against the virtual retropad. If that's indeed the case, DO NOT go into settings > input > retropad binds and start mucking around with stuff to move around your N64 inputs. This WILL cause you a bunch of grief.
By default, the N64-focused libretro cores assign their inputs to mimic Nintendo's own mapping for N64 games on modern controllers, as exemplified by the Wii Classic Controller mapping for the Wii Virtual Console games. That is, the C-buttons are mapped to the cardinal directions of the right-analog stick, while the R2 button (right trigger) is used to trigger the "C-button Mode," whereby the C-button functions move up to the corresponding four face buttons (ABXY) as long as the button is held down.
Most games only used the C-buttons for things that don't require precision, such as camera control in Super Mario 64, and many others used them for precise inputs, but only very briefly, such as hammering out ocarina tunes in The Legend of Zelda: Ocarina of Time.
Both of these cases are covered very well by the default mapping, and I would strongly recommend everyone with a modern controller give this a shot before going crazy in the remap menu. It takes a few minutes at most to get the hang of it, and it's very comfortable and effective once you do.
There is a small handful of games that don't work well like this, such as Killer Instinct Gold, which treats the C-buttons as first-class citizens to have the traditional 6-button fighting game layout. In this case, having anything on the right-analog is awkward/imprecise, and it's simply not feasible to require tapping R2 to trigger "C-button Mode" on/off throughout a complex combo. For these games, we have the "Independent C-button Controls" core option, which takes the right-analog stick out of the picture entirely and removes the R2/C-button Mode function. Instead, all of the face buttons are moved over to digital inputs on the retropad, so you can move them around as you see fit in quick menu > controls > port 1/2/3/4 controls.
Modern 6-button Controller <-> N64 Pad
Recently, there have been a number of gamepads to hit the scene with 6 face buttons, usually in the form of "fightpads" for fighting games and/or based on Saturn/Mega Drive 6-button pads. I've seen a lot of people claim incorrectly that these pads are "incompatible with the retropad because of the 6 face buttons vs the retropad's 4." Again, buttons is buttons. As long as you have all of the physical gamepad's buttons mapped to something on the retropad, you can move the core's functions around on them however you like, and the "Independent C-button Controls" option makes everything nice and straightforward (i.e., none of the confusing "B / C-Left" nomenclature, which, for the record, is describing the regular mapping alongside the R2/"C-button Mode" mapping).
Original N64 Controller <-> N64 Pad
This one should be the most intuitive one to map, but it's actually the least. There are two main ways to handle it, and you'll have to decide which way you want to go at the outset: stick with the default core mapping and manage our retropad <-> N64 controller mapping accordingly, or use "Independent C-button Controls" core option, and the retropad <-> N64 controller mapping is nebulous (you just have to keep straight whatever you choose).
Default Core Mapping
If you go with the former (i.e., no "Independent C-button Controls" option), we can crib from one of our existing autoconfig profiles to save ourselves the trouble of looking up the retropad <-> core function mappings manually:
input_b_btn_label = "A"
input_y_btn_label = "B"
input_start_btn_label = "Start"
input_up_btn_label = "D-Pad Up"
input_down_btn_label = "D-Pad Down"
input_left_btn_label = "D-Pad Left"
input_right_btn_label = "D-Pad Right"
input_l_btn_label = "L"
input_r_btn_label = "R"
input_l2_btn_label = "Z"
input_l_x_plus_axis_label = "Joystick Right"
input_l_x_minus_axis_label = "Joystick Left"
input_l_y_plus_axis_label = "Joystick Down"
input_l_y_minus_axis_label = "Joystick Up"
input_r_x_plus_axis_label = "C Right"
input_r_x_minus_axis_label = "C Left"
input_r_y_minus_axis_label = "C Up"
input_r_y_plus_axis_label = "C Down"
So, head over to settings > input > retropad binds > port 1 controls, hit "Reset to Default Controls," then hit "Set All Controls." This will guide us through the mapping process, offering up a retropad button, and then listening for which physical button we want to map it to. Use the button label list above as a guide for what to press when. The only trick here is that I recommend also mapping retropad-A to N64-B (i.e., you're mapping both retropad-A *and* retropad-Y to the same N64-B button), which will be useful for navigating RetroArch's menus. It's not vital, but it's a nice convenience, especially if you won't have a keyboard handy when you're playing.
Once you're finished mapping, hit "Save Controller Profile," and then hit "Reset to Default Controls" again.
Just for the record: if you don't 'Reset to Default Controls' before mapping, any manual mappings you've made previously will interfere with your profile, and if you don't do it afterward, your crazy N64 mappings will screw up any subsequently loaded autoconfig profiles, since manual mapping supersedes auto-mapping in RetroArch
Independent C-button Controls Enabled
This method is pretty straightforward, but it requires you to be on top of translating your N64 buttons to retropad inputs, so I recommend taking notes as you map things, since you're going to need them when you're moving things around in quick menu > controls > port 1 controls.
Just like above, head over to settings > input > retropad binds > port 1 controls, hit "Reset to Default Controls," then hit "Set All Controls." As it goes through the available retropad buttons, map them wherever you like, just make sure you keep track of what is going where. When you're all finished, hit "Save Controller Profile," and then hit "Reset to Default Controls" again.
Next, load up your core plus content, hop into the "Quick Menu" and scroll down to "Controls," then "Port 1 Controls." This is where you can move your core functions (the column on the right) around on the gamepad/retropad buttons (the column on the left) without messing up your global/retropad inputs that affect everything else (other cores, menu controls, etc.). The only trick here is that, unless you've added button labels to your autoconfig profile, you're going to see the retropad labels in the column on the left instead of your physical N64 pad's button labels, so you'll want/need to consult your notes to see what is mapped where--just like the Steam/button label example at the start of this piece.