Update: July 26th 2011 Putty 0.61 has been released. My patch is incorporated in this release, and is listed as part of the release note.
From time to time, I would create a SOCKS proxy with SSH to secure my FTP or HTTP transfers. PuTTY is by far my favorite SSH client in Windows because it is free, lightweight, and supports advanced feature such as SSH connection over HTTP proxy.
Last week, when I was sending some data over to a FTP server through a PuTTY’s SOCKS proxy, I noticed something strange. Filezilla (FTP client ) indicated that all file transfers are 100% completed, yet the uploaded files were almost always incomplete. I retried the same upload numerous times, but would always result in almost completed file.
Here's the original file size and uploaded file size. Notice that they are not quite the same.
After trying different software combinations, I eventually figured out that the bug is originated from PuTTY. And some internet research shows that the bug has been around since 2003!
The Why and the How
I was bored, so I decided to download the source code of the PuTTY nightly snapshot and started a debug session. (Yes, real programmers debug FOSS after dinner for entertainment.)
Since I was completely new to the source code and SSH2 protocol, the bug certainly wasn’t obvious. I spent most of the first night setting up breakpoints and tracepoints to understand the logic path.
By the second night, I got a good handle on the way the sockets are created, and started reading the Connection Layer (RFC 4254) of the SSH2 protocol.
By the third night, I found the reason for the port-forwarding data corruption.
When PuTTY forwards a port, it sets up a TCP server at the local endpoint to receive the traffic, and then it sets up a TCP client to forward to traffic to the SSH server.
In terms of implementation, this is a three-step process for PuTTY.
1. PuTTY receives data from the local endpoint. In my case, it is Filezilla.
2. PuTTY buffers up the data in its own queue – the buffer chain.
3. Then PuTTY will send the data to the SSH server, with respect to the SSH2 sliding window protocol. The SSH2 window size is small – only 16 kB (no idea why). Therefore, PuTTY would often fill the window quickly, and buffer up the data in the buffer chain. The data will be only sent as soon as the SSH server opens up the window through a SSH2_MSG_CHANNEL_WINDOW_ADJUST message.
Unfortunately, step three has a bug. When the local endpoint closes its connection to PuTTY, PuTTY would immediately close its channel to the SSH server without fully sending all its data. So the last set of channel adjustment message are ignored, and led to data being stuck in the buffer chain.
Because the channel is closed prematurely, the last set of window adjustment messages are ignored.
The Fix
I modified PuTTY’s SSH2 channel closing sequence to keep the channel open until the buffer chain is completely emptied. For debugging, I also added some events in PuTTY’s event log to indicate the number of bytes PuTTY’s trying to send before closing the channel.
PuTTY event log to record the shut down scenario.
This modification works like a charm for me. I tested hundred of GB of small file uploads, and not a single file was corrupted.
I submitted the patch to PuTTY development team, and it was accepted by Simon Tatham – the development lead. The patch was checked into the PuTTY’s trunk directory as rev. 8971, and is built into the PuTTY nightly snapshot.
svn://svn.tartarus.org/sgt/putty rev. 8791
Final Thoughts
You can download my patch here if you are interested in the details.
PuTTY’s source code is clean and self explanatory. It is written by good programmers.
Now I have something to brag about in front of my nerdy co-workers and friends. 🙂
ASKLDJD 
Alan – this is great! Now someone just needs to update http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/portfwd-corrupt.html showing this as fixed in the nightly build.
I also hope that PuTTY will have a new release version soon. 0.60 is over 3 years old, and too many people don’t like running non-release versions, e.g. the nightly snapshots. Having a new release would also be beneficial to other related projects, e.g. puttycyg: http://code.google.com/p/puttycyg/issues/detail?id=30 .
WOW! Major kudos!
Nice!
I am sorry to let you know, that snapshot 2010-07-28:r8971 still doesn’t work.
I am using PuTTY to local port forward to HTTPS M0n0Wall (BSD based firewall) interface. It’s true that on the remote linux machine this port is again locally forwarded to another machine.
But if I use SecureCRT on my local machine, port forwarding works with no problem, on PuTTY some HTML pages are only partially downloaded.
Any clue on the issue?
Event Log contains entries like this:
2010-07-29 00:06:19 Opening forwarded connection to 127.0.0.1:4431
2010-07-29 00:06:19 Nothing left to send, closing channel
2010-07-29 00:06:19 Forwarded port closed
2010-07-29 00:06:21 Opening forwarded connection to 127.0.0.1:4431
2010-07-29 00:06:22 Nothing left to send, closing channel
2010-07-29 00:06:22 Forwarded port closed
FYI, I just tested this again with local port forward.
So I have PuTTY local port forward with
src port = 4431
dest port = 192.168.1.125:40099.
I have a TCP server listening on 192.168.1.125 port 40099 for data. Then I use a TCP client to connect to 127.0.0.1:4431, and sent couple large (5 MB) and small (200 bytes) files over.
None of the files were corrupted.
2010-07-28 22:07:08 Opening forwarded connection to 192.168.1.125:40099
2010-07-28 22:07:08 Nothing left to send, closing channel
2010-07-28 22:07:08 Forwarded port closed
2010-07-28 22:07:20 Opening forwarded connection to 192.168.1.125:40099
2010-07-28 22:07:37 Forwarded port pending to be closed : 10231 bytes remaining
2010-07-28 22:07:37 Forwarded port closed
2010-07-28 22:07:44 Opening forwarded connection to 192.168.1.125:40099
2010-07-28 22:07:52 Nothing left to send, closing channel
2010-07-28 22:07:52 Forwarded port closed
If possible, I would like to see a similar test with your setup. Since you are doing multiple forwarding, the SSH windowing scenario might be a bit more complicated.
Furthermore, I only fixed the bug with SSH2. I did not modify SSH1 logic in PuTTY.
… Alan
Uros,
This looks like a different bug.
I added the string “Nothing left to send, closing channel”. This implies that PuTTY has waited for all the window adjustment message to come back, and there are nothing left to send (to PuTTY’s perspective).
Maybe there is another problem here. Can you prove that PuTTY has in fact received all the data? You can turn on SSH Packet and Raw Data logging to log every packet received and sent by PuTTY.
If the packets are received, and forwarded by PuTTY, it might be a socket issue.
If the packets are received, but aren’t forwarded by PuTTY, it might be that the bug I worked on still exist, but in a different manner.
Let me know. I am interested.
Thanks
… Alan
I am not using SSH1, only SSH2 🙂
I switched on SSH packets and raw data logging, but all I could see is encrypted traffic, since I connected to the SSL encrypted HTTP (HTTPS).
I changed HTTPS to HTTP, so I could see raw HTML data and noticed the following in the putty.log:
Incoming packet #0x99c, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
…
Incoming packet #0x99c, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
…
Incoming packet #0x9ae, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
…
Outgoing packet #0x878, type 2 / 0x02 (SSH2_MSG_IGNORE)
00000000 00 00 00 00 ….
Outgoing packet #0x879, type 98 / 0x62 (SSH2_MSG_CHANNEL_REQUEST)
00000000 00 00 00 01 00 00 00 22 77 69 6e 61 64 6a 40 70 …….”winadj@p
00000010 75 74 74 79 2e 70 72 6f 6a 65 63 74 73 2e 74 61 utty.projects.ta
00000020 72 74 61 72 75 73 2e 6f 72 67 01 rtarus.org.
Outgoing packet #0x87a, type 2 / 0x02 (SSH2_MSG_IGNORE)
00000000 00 00 00 00 ….
Outgoing packet #0x87b, type 93 / 0x5d (SSH2_MSG_CHANNEL_WINDOW_ADJUST)
00000000 00 00 00 01 00 00 26 f8 ……&.
…
Incoming packet #0x99c, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
…
Incoming packet #0x99f, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
…
Outgoing packet #0x886, type 2 / 0x02 (SSH2_MSG_IGNORE)
00000000 00 00 00 00 ….
Outgoing packet #0x887, type 93 / 0x5d (SSH2_MSG_CHANNEL_WINDOW_ADJUST)
00000000 00 00 00 01 00 00 99 08 ……..
Incoming packet #0x9a4, type 100 / 0x64 (SSH2_MSG_CHANNEL_FAILURE)
00000000 00 00 01 01 ….
It looks like browser stops receiving data after one of SSH2_MSG_CHANNEL_FAILUREs.
I do not know exactly why browser stops receiving data (port is closed either on server or client side or something else).
I will try to find out answer to this problem, after sniffing exactly what is received by the browser.
Thanx again for your help.
High 5! Gooooo Nerds!
I have tested this in Windows XP 32-bit and it works perfectly, without a problem.
It seems this is either Windows 7 or mere 64-bit OS bug.
Can you test it on some Windows 7 64-bit edition?
I am running Window 7 64 bit. However, PuTTY is running in 32 bit mode.
This is really strange.
I have now setup PuTTY (32-bit original version) on my Windows 7 64-bit machine.
I enabled “Local ports accept connections from other hosts” and ran web client on remote machine, so I was able to sniff traffic between web client and PuTTY Local port (forwarded connection).
I also enabled SSH packet logging in PuTTY.
Not every time, but in some cases, which is extremely strange, PuTTY ended forwarding HTML data to web server prematurely (in the middle of HTTP stream), which I verified using WireShark. HTML is approximately 50 kB in size. Data sent from PuTTY to web client is fragmented. PuTTY SSH packet logging revealed that complete HTML is received in all cases.
I therefor assume, that there is some kind of port forwarding bug in PuTTY. Maybe some indexes in buffer handling 😛
Uros,
I find it surprising that the packets are sent out of order. PuTTY is single-threaded, so I don’t expect race conditions. It uses an async socket with select(), so packets are received, and queued and sent in a FIFO manner.
However, the buffer queue is written in a custom 234 tree (no idea why).
Also, what type of port forwarding are you using? Is it Dynamic, Local or Remote?
I am trying to set up a test bench to recreate your scenario. It is probably going to take a little while.
Thanks for the info.
… Alan
Just a thought. What if there is a problem when forwarded connection (to the web browser) still hasn’t received all the data, but SSH connection to the original server is already broken (closed)? What if PuTTY then closes the connection to the forwarded connection (web browser) as well, instead of delivering the rest of the data to it.
I am using Local port forwarding 443 -> 443.
Thanks for your help.
This is possible. The original developer has mentioned that PuTTY do not support half-close socket connection. So at the moment, if one side closes the connection, PuTTY brings down the entire SSH channel. Since TCP is a full duplex transmission, it is not the correct implementation if all data must be reliably sent.
I also just realized that my FTP test is not adequate for your scenario. My FTP forwarding is mostly data going outbound, whereas your HTTPS data is coming inbound.
Let me see if I can reproduce the problem here easily (I might not). I will start looking into how hard it is to add half-close TCP sockets to PuTTY.
Thanks
… Alan
Dear All,
Interesting post.
I have a similar problem with putty.
I need to open an ssh v2 session from my laptop to open a tunnel from my pc to a linux box running dropbear.
The box is a enigma2 OS running dropbear (ssh server) and an http proxy.
My laptop run on linux ubuntu11.04 with putty Development Snapshot 2010-12-08
I would like to open a session on a remote box and define a tunnel like 8080:127.0.0.1:8080
On the remote linux box the TCP port 8080 is the HTTPProxy.
The goal is to bypasse a firewall (in a corporate network) and use the remote proxy on linux box.
I can open my ssh session, but when I’m going on a web page, it begin to load but putty crash with error: SSH2_MSG_CHANNEL_FAILURE for half-open channel 258
I also try a win32 version of putty running within CrossOver (Wine) but I have exactly the same issue
For Information
In the other hand, I have another linux server (but a real linux on a dedicated server, running squid (a real squid proxy) and I can build a tunnel toward this server with the same putty client without any problem. Everything works fine..
Hope this Help
Franck (from France)
Same here as Frank, my SSH server is dropbear on openwrt, I can tunnel a RDP session to my home windows, but can’t tunnel a http session to my home web server
OK, I tried 3 dropbear version ssh, openwrt and tomato doesn’t work, get the SSH2_MSG_… error, but ddwrt version works.
Interesting. I use DDWRT dropbear at home, and I haven’t seen this problem before. Are you guys using the latest version of PuTTY?
Same issue here after installing 0.61 putty. Windows XP SP3 32-bit connecting to Asus wl-500g with custom firmware (linux based) running dropbear 0.50. Tunneling port 1080 (socks) to a srelay 0.4.6 instance.
(X) Disconnected: Received SSH2_MSG_CHANNEL_FAILURE for nonexistent channel ###
I still have this problem when using putty as a tunnel.
As of 11/17/11 the bug is still there. I cannot use putty´s port forward feature as it simply doesnt work for http https tunneling. I am using the nightly build. Does anyone have a portable alternative for putty which implements this feature correctly?
Sorry to inform you guys, but even version 0.62 is buggy as fuck.
if you want properly written implementation, you need to check
Tunnelier (Bitvise) > Putty crap
cheers
I had the same issue. Yes, the 0.62 version is still ‘buggy’. But the nightly snapshot works fine…
Wow, I’ve had trouble tunneling to a test web server (ports 80 and 443) for over a year now using putty. When I tried it with Bitvise (per the above comment), finally, it works! Before that, I got strange inconsistencies that seemed to be random bits dropping off.
Thank you!
Hello.
Bug still exist, and I can easly confirm this.
When you are using dynamic (SOCKS4) port forwarding and remove connection is closed rapidly (fast server and fast connection) putty
fails to deliver last chunk of data. It is even worse compared to 0.60
where I loose few 512 blocks compared to many 512 blocks in 0.62.
Test case (using simple socks4 connect tool):
server side: cat | nc -l -p 5000 -q 1
client side: connect -S localhost localhost 5000 > file
you will see that file is incomplete. remove -q 1 from netcat
and then give it a few secs to setle, file will (should) be completed.
It needs to be fixed 🙂
Good news. Seems problem has been fixed in latest development snapshot
(2012-11-08:r9690). I’ve done my test cases and now 100% of them pass
in both directions 🙂
Where can I get a binary that has the issue fixed? I have not been able to find it anywhere!
Bitvise works great! Latest Putty did not work for me, I had the same SSH2_MSG_… error
I had lots of putty fatal errors: server sent disconnect message type 2 (protocol error)
“received oclose for nonexistent channel 66571288.”
I used putty 6.1
We use a lot of port fowarding with SSH tunnelling.
So I decided to use Putty 6.2 beta. This specific error disappeared, but after some time I decided to go back to putty 6.1 because the result of 6.2 was worse. This time, lots of time nothing happened at all, or my broser lost network connection completely, refreshing all the time sometimes solved it but upon other occasions browser hangs
I use PuTTY 0.62 x64 (from Splunk) to tunnel a connection to a CentOS server running Squid. I am still seeing partial page loads, and corrupt downloads. I have upgraded Squid, and played around with compression and keepalives, but to no avail. This same issue was present in 0.60.
I had thought the incomplete page loading was a Squid problem to begin with, but it also happens if I tunnel local port 80 to remote :80, and try looking at the web pages. Refreshing them a few times usually works. The Apache server at the remote end shows the same content length regardless of whether or not the page loads completely in the browser.
Hi Alan,
How can i get a copy of your patch file? its seems i cannot access the URL for patch file.
Thanx
The patch was checked into the Putty SVN trunk several years ago.
The SVN link is here: svn://svn.tartarus.org/sgt/putty
The rev you are interested in is 8971.
Revision: 8971
Author: simon
Date: Sunday, July 04, 2010 6:53:53 PM
Message:
Patch from Alan Ning (somewhat polished by me): introduce a flag
called ‘pending_close’. This deals with the situation in which we’re
forwarding a port, have received and locally buffered some data from
the local endpoint but not yet been able to pass it down the SSH
connection due to window limitations, and then the local endpoint
closes its socket. In this situation what we’ve been doing until now
is to immediately send SSH2_MSG_CHANNEL_CLOSE, causing the data
still in our local buffer to be lost; now we instead set the new
flag, which will remind us to send SSH2_MSG_CHANNEL_CLOSE _after_ we
empty our bufchain.
Should fix at least one manifestation of ‘portfwd-close’, though I
don’t know if it’s the cause of all the reports we’ve ever seen.
—-
Modified : /putty/ssh.c