Ok, so we finally got around to building 'megatools-1.11.0-git-20210505.tar.gz'
on Windows msys2/mingw64-gcc.

By the way, it was our first time using meson/ninja and it was surprisingly much
easier to get working than using GNU autotools, which from our previous
experience, was very error-prone on Windows cygwin64-gcc or msys2/mingw64-gcc.

We continued to face that final error message (after 100.0% uploaded) - and so,
as did exactly like you suggested - inserted a do-while loop to re-do that final
http_post until it succeeded - and inserted a g_printerr, g_usleep statements in
the loop for debugging and non-DDoS respectively.

This seemed to fix the problem - rarely - about 1 out of 5 times - the first
http_post works to finalize/complete the upload, but with this do-while loop
hack, the second http_post works to finalize/complete the upload - about 4 out
of the remaining 5 times.

Again, like you said, this bug only seems to show itself when using 'tor' - and
we confirmed that it happens due to using 'tor'.

We also use 'obfs4proxy' to obfuscate our 'tor' connection and prevent
ISP/middleman from knowing we are using 'tor'.

And prior to fixing this bug with this do-while loop hack, we tried both with
and without using 'obfs4proxy' - and the bug remained exactly the the same
whether or not 'obfs4proxy' was used - so, it has nothing to do with
'obfs4proxy' but exclusively to do with 'tor'.

As to why this do-while loop hack was needed to fix this bug after many, many
months of successfully (without any error) uploading to our megatools account
using 'tor' without this hack, we believe that one of the following is true:

(1) unlikely: Either ISP/middleman does not like the bandwidth we are using by
uploading large files which takes a long time (especially over tor) - and so,
they are sabotaging the final connection.

(2) maybe: The 'tor' exit nodes do not like tor-users that are using up
limited-bandwidth for uploading large files to mega.nz for long periods of time
- and so, they are dropping the final connection.

(3) likely: mega.nz is trying to curb 'tor' usage or limit the
amount/size/volume of uploads from 'tor' connections, and so, they have
introduced this final block to deter/discourage tor users.

We don't use 'tor' for no reason - 'tor' users include journalists,
whistleblowers, activists/protesters, political dissidents, oppressed
minorities, military/top-secret-government-employeees - and so, all of these
people need to use 'tor' as a legitimate method to avoid potentially-dangerous
surveillance that might be harmful to them if their enemies knew what they were
doing online.


Instead of uploading a patch to you, we will simply paste as follows, a GNU
'sed' command that patches the code to hack this bug - the patched file is
'lib/mega.c' and the original file is 'lib/mega.c.sed' so, that you can compare
the two:

/usr/bin/sed --in-place=.sed 's/ GString \*res_str = http_post(s->http, url,
req_node, strlen(req_node), &local_err);/ GString\* res_str = NULL;\n gint
number_of_httppost_retries = 0;\n do {\n local_err = NULL;\n res_str =
http_post(s->http, url, req_node, strlen(req_node), \&local_err);\n
++number_of_httppost_retries;\n if (number_of_httppost_retries > 1) {
g_printerr("ERROR: Having to re-do http_post due to error: %d times...\\n",
number_of_httppost_retries); }\n g_usleep(250000);\n } while (res_str ==
NULL);\n/g' ./lib/mega.c


This hack patch should only modify 1 line into 10 lines:

$ /usr/bin/diff --report-identical-files ./lib/mega.c ./lib/mega.c.sed
1417,1426c1417
< GString* res_str = NULL;
< gint number_of_httppost_retries = 0;
< do {
< local_err = NULL;
< res_str = http_post(s->http, url, req_node, strlen(req_node), &local_err);
< ++number_of_httppost_retries;
< if (number_of_httppost_retries > 1) { g_printerr("ERROR: Having to re-do
http_post due to error: %d times...\n", number_of_httppost_retries); }
< g_usleep(250000);
< } while (res_str == NULL);
< 
---
> GString *res_str = http_post(s->http, url, req_node, strlen(req_node),
&local_err);




Thanks!





> On Monday, 18. October 2021 2:56, Ondřej Jirman
[/webmail/send?to=megatools@megous.com] wrote:

Hi,

On Thu, Oct 07, 2021 at 04:46:52AM +0000, YruNlsr6371bDg wrote:
> Hello,
>
>
>
> We tried the following using 'megatools-1.11.0-git-20210505-win64' (but we
> presume the same error message will apply to all recent megatools versions
> across all platforms).
>
> We run 'megaput' with the following command-line arguments to increase
> verbosity/debugging:
>
> megatools.exe put --ignore-config-file --no-ask-password
> --debug=http,api,fs,cache,tman --proxy=socks5h://127.0.0.1:9050
> --username=************************ --password=***************************
>
'************************************************************************************************************************************'
>
>
>
> After many months of successfully using megatools to access/upload to our Mega
> account, since approximately 5 days ago, we are now not able to complete
upload
> larger files to our Mega account using megatools due to end-of-upload error:
> "error:1421C0F8:SSL routines:set_client_ciphersuite:unknown cipher returned"
>
> (We also tried deleting the cache-files/temporary-directory, but that did not
> fix the problem either.)
>
> This issue might be due to a change in the way Mega servers handle the upload
of
> large files - and thus, the HTTP post fails due to the the initial connection
> being too old ("Too old connection (***** seconds), disconnect it"):
>
> If so, the solution might be to keep the original connection alive - even if
it
> is a long upload, so that reconnection is not required - this is probably what
> results in the failed upload, or at least provide a command-line option to
keep
> keep the original connection alive such as "--keep-alive" similar to the
OpenSSH
> "TCPKeepAlive" option.
>
> We also looked at the source code from
>
'https://megatools.megous.com/builds/experimental/megatools-1.11.0-git-20210505.tar.gz'
> to try to debug this problem, but this seems to be a curl library error caused
> by disconnecting/reconnecting the "too old" original conneciton - which the
Mega
> servers might not be allowing any more.
>
>
> The final error message received after reaching 100.0% is uploaded is posted
as
> follows (with any private information redacted).
>
> In all cases, even when we repeatedly try to upload large files, the upload
> correctly progresses from 0.00% to 100.00% - but when reaching 100.00%, it
> returns the following text within quotes:

Mega is likely just not caring for connection to API server. HTTP is stateless
protocol. It's weird that connection fails, but give that this is over tor,
it may not be mega.nz failing but something in between.

Maybe re-trying the API calls in case of this kind of failure may help.

I'd suggest you try to modify the code to add a bunch of retries to the final
API call that finalizes the upload.

kind regards,
o.

> "
> .
> .
> .
>
*********************************************************************************************************************************:
> 100.00% - done *.* GiB (avg. ***.* KiB/s)
> 1 left intact
> * Found bundle for host **********.userstorage.mega.co.nz: 0x***********
> [serially]
> * Can not multiplex, even if we wanted to!
> * Re-using existing connection! (#0) with proxy 127.0.0.1
> * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0)
> > POST
>
/ul/**************************************************************************************/**********?c=****************
> HTTP/1.1
> Host: **********.userstorage.mega.co.nz
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Megatools (1.11.0)
> Content-Type: application/octet-stream
> Content-Length: *******
>
> * We are completely uploaded and fine
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Type: text/plain
> < Content-Length: *
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> * Found bundle for host **********.userstorage.mega.co.nz: 0x***********
> [serially]
> * Can not multiplex, even if we wanted to!
> * Re-using existing connection! (#0) with proxy 127.0.0.1
> * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0)
> > POST
>
/ul/**************************************************************************************/**********?c=****************
> HTTP/1.1
> Host: **********.userstorage.mega.co.nz
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Megatools (1.11.0)
> Content-Type: application/octet-stream
> Content-Length: ******
>
> * We are completely uploaded and fine
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Type: text/plain
> < Content-Length: *
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Type: text/plain
> < Content-Length: *
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Type: text/plain
> < Content-Length: *
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Type: text/plain
> < Content-Length: *
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> * Mark bundle as not supporting multiuse
> < HTTP/1.1 200 OK
> < Content-Length: **
> < X-Content-Length: **********
> < Content-Type: application/octet-stream
> < Access-Control-Allow-Origin: *
> < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> < Access-Control-Max-Age: 86400
> < Cache-Control: no-store
> <
> * Connection #0 to host 127.0.0.1 left intact
> -> [{
> "a": "p",
> "t": "********",
> "n": [{
> "h": "************************************************",
> "t": 0,
> "k": "*******************************************",
> "a":
>
"************************************************************************************************************************************************************************************************",
> "fa": null
> }]
> }]
> * Too old connection (***** seconds), disconnect it
> * Connection 0 seems to be dead!
> * Closing connection 0
> * Trying 127.0.0.1:9050...
> * SOCKS5 connect to g.api.mega.co.nz:443 (remotely resolved)
> * SOCKS5 request granted.
> * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#1)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * SSL re-using session ID
> * error:1421C0F8:SSL routines:set_client_ciphersuite:unknown cipher returned
> * Closing connection 1
> ERROR: Upload failed for
>
'************************************************************************************************************************************':
> API call 'p' failed: HTTP POST failed: CURL error: SSL connect error
> SAVE CACHE: {
> "version": 4,
> "last_refresh": **********,
> .
> .
> .
> "