http://www.lua.org/manual/5.1/ is the official reference for looking up lua things that are part of the base language and not added by Stepmania.
http://www.lua.org/manual/5.1/manual.html#3 explains why stack indices can be negative.
Positive indices are usually used to access parameters, because they are constant, and not relative to the top of the stack.  Negative indices are usually used to access recently pushed things.
AddVertex assumes that it has been passed one table, which is on the top of the stack.  The top of the stack is also stack index -1, or the stack index returned by lua_gettop.
Since nothing else has touched the stack before the function runs, lua_gettop returns 1, so it would also be safe to use 1 instead of -1 when calling lua_objlen and set vert_table_index to 1.
If three tables were passed, they would be at stack indices 1, 2, and 3, or -3, -2, and -1.
vert_table_index could not be safely set to -1, because the top of the stack moves whenever something is pushed or popped.
lua_pushnumber is used to push the value that is going to be used to index into the table that was passed.
lua_gettable pops the table index and pushes a copy of the element at the table index onto the top of the stack.
In this case, it was a table that was just pushed onto the top of the stack, so its stack index is stored so its elements can be accessed.
lua_rawgeti indexes into the table and pushes the value at the table index onto the stack, then lua_tonumber is used to set the thing the number is meant for.
lua_pop is used to pop the four things that were pushed onto the stack, to avoid a stack overflow.
So, if three tables were passed, with position, color, and coord data, indices for all three of them would need to be stored.
For ease of use, color and coord data should be optional.  lua_type can be used to check the type of a value on the stack, and whether it exists.
For faster processing (using a single loop for all three tables), all passed tables could be forced to be the same size.
Each time through the loop, for each table, the steps would be:
1. Push the stack index of the next piece of data.
2. Retrieve that piece of data.
3. Retrieve its elements to set the properties of the vertex.
4. Pop the things that were pushed onto the stack.
Note that the stack index has to be pushed again for each table, because lua_gettable pops it off the stack.