need to overwrite RIP but the address has \x0a in it

Alas, your domain is mega specific. However, all fgets does is read a string from a FILE pointer. I am not too sure how you are getting an overrun as fgets specifies the number of bytes to read and if it hits the EOF marker before reading the full buffer it returns. The only thing I can suggest is you are asking to read >50 bytes into an allocated buffer of 50. If you are getting overruns, i.e. you are splatting over random areas of memory after 50 bytes, then there is no way you can recover. It like spilling coffee over your shopping list. Game over.

Good luck!

Stack overflow would have been your best option. I'm not great with assembly, it is on my to do list though.
As far as I know, your don't overwrite RIP as its the instruction pointer.
But I have no idea how you're getting a buffer overflow when reading a string, we'd have to see the whole code to figure that one out.

Sorry just to clarify that normally in a simple buffer over flow exploit, the aim does tend to be to overwrite rip (or eip in x86) as you're meant to overwrite it to an address that points to your shell code, either in an env variable or on the stack. The code itself had something like char buff[64] (so anything over 64 overflows eventually hitting rip).

Doesn't matter now anyway as this was the incorrect method. I was half right by overflowing the buffer. But didn't realise I could change the seed to srand() and so predict the "random" choice of rock paper and scissors. Never mind

Were you trying to create a buffer overflow?
I'm lost in what you were trying to do exactly.

Ill explain the entire task.

so, you are given a binary written in C, its basically a rock paper scissors game. you are also given a server to connect to that, once connected, runs the game and asks for input, but more on that later.

The rock paper scissors (RPS) program runs as such.

1) ask user for their name
2) greets the user eg hi jared44
3)the program then "randomly" chooses r p or s and asks for your choice.
4) if you win, you get 1 point, and the game starts again.
5)if you lose, the game ends.

You have to beat the program 50 times in a row. when you beat the game, it then opens a file, reads the content (the flag) and then prints that to the screen.

Now to the exploit. It turned out that the buffer holding the players name was 50 bytes. Imagine char name[50]. If you enter 128 A's then you get a seg fault (cannot access memory at 0x41414141414141), so that means at 128 (or less) bytes, RIP has been overwritten.

All seems well, now all i had to do was find the exact number of bytes to fill the buffer with, minus the number of bytes for a return address so jump execution to. simple enough. There was a section in the disassembled code (seen via gdb), that was something like jle rbp-0x4 0x32 (which means, if rbp-0x4 is less than 0x32 jmp to ....) so a logical choice would be to overwrite RIP to the address just after this check, avoiding the need to beat the game 50 times.

Now, the problem. the memory address just after (and a fair few addresses before and after) this section was something like 0x0000400a81. You see that \x0a, thats a new line feed byte. So, lets say i started the program something like this at the cmd line:

$ python -c 'print "A" * 120 + "\x81\x0a\x40\x00\x00\x00"' > ./rps_game

what would happen is the game would read in 120 A's then \x81, then \x0a, but then it would create a line feed and "ignore" the rest of the input, thus not properly overwriting the memory address. This is what i was struggling with.

The real solution.

So it turned out, the game was using a seed for the random number, so it was choosing a number between 0,1,2. where 0 = r, 1 =s etc And it just so happened that the seed was located around 16 bytes after the name buffer in memory. So what you needed to do was overflow the name buffer, with say 66 A's. and then write a small script that seeded a random number generator with the same seed that you will overwrite the rps game with. This means that your answers will exactly match the vulnerable games.

all that was left to do then was connect to the server, overwrite the buffer, and then keep matching the games answers, which in the end was extremely simple

So the random seed integer was after the name buffer, but before any memory that was not in the address space of the process. Like so:

struct game
{
char name[50];
//16 bytes worth of some data...
int random_seed;
};

So that's very easy. But your original attempt makes no sense. You were trying to set RIP when the only data you could inject was the name input? Makes no sense. Unless you know the struct was on the stack. Then there could be a return address in there that you could overwrite. But if this exe is on a server, you wouldn't know the address of the function that displays the "you win" message anyway; and there would be no way to inject your own code as that would sit on the stack (read/write but not execute permissions).

So, if you know the address of the "win" section of code, and you can see the overflow on the stack, it's pretty straightforward to overwrite rip and Change the execution.

By the bold part, I assume you mean where the struct ends, and rest of the stack continues? Thing is, even if you have the source-code and know exactly how it was compiled, you defo wouldn't know the address of the "win" function

Actually I just remembered something. On most OSs the stack is upside down in memory. So the injection point would be from a HIGHER memory address than the rest of the stack:

[high memory]
start of stack
injection point (injecting towards higher memory)
nearest return address
RSP
end of stack
[low memory]

Although it depends on calling convention and compiler used, as well as in which scope the struct was declared in (assuming this was written in C).

In fact you can see exactly where every instruction is in memory using gdb.

But not on the remote server.

actually you could do this, this person did what I did, except he persevered with finding an address to overwrite RIP with, whereas the rest of us just over wrote the seed.

https://jkrshnmenon.wordpress.com/2015/09/09/mmactf15-rps-write-up/

Some of these pwnage games go to great lengths to disable Address Space Layout Randomisation, memory security tagging etc. etc.

I have no problem believing that simply examining the binary was sufficient in this case, even though it never normally would be.

How did he know that the address of the instruction on a remote server was at 0x4008b4? If he knew it, then yea, it's easy; and the solution (to jump to an address which jumps to an address, is a good way of doing it).

Some of these pwnage games go to great lengths to disable Address Space Layout Randomisation, memory security tagging etc. etc.

I have no problem believing that simply examining the binary was sufficient in this case, even though it never normally would be.