so let me tell you a story of how i brought the entire site to its knees with infinite text (or what i will now refer to as textageddon)
due to the db errors yesterday, i added code that watched for an error, and then performed a rollback() if the request caused an error. this was the beginning of textageddon.
I woke up today and noticed shit was still breaking, and decided just to add a check to EVERY request that rolled back the request if the request failed due to a db error. This was the equivalent of pouring gasoline onto the fire, will get to why in a bit.
the important part of how comment/post/message/etc are rendered WAS as follows:
for c in comments:
c.text = pseudo_markup(c.text)
c.text then can be rendered safely.
what caused the disaster is that since c is an sqlalchemy object that is stored in the session, and the .text was set as the result of the markup function, if the request errored out, and was rolled back, the c with the NEW c.text would be committed to the db because rollback () would commit the new modified object, unaware/uncaring that it had changed.
this meant that if any text containing html that was escaped, was being commited to the database in escaped form on rollback.
this ALSO meant that the error would NOT occur in the test environment that i had setup, from the DB dump of the live site. since the text in records was already messed up from the very start, it couldn't grow any larger past the 20,000 character length on rollback().
wnewlad also downloaded the db dump, and couldn't replicate what was happening... this is why.
once i FINALLY figured out what was happening, and more importantly WHY it was happening, i changed all comments/text/messages to store the html_escaped text as an attribute named x.new_text instead of x.text, this means on rollback, it will be discarded instead of committed.
i still have to make sure every place which presents htmlescaped user input uses the new attribute name, i pushed out a patch as quickly as possible to get shit running again.
since the DB was now completely fucked up in who knows how many places, I restored from a db dump I made yesterday before going to bed.
and that is the story of textageddon. the site was so fucked up that not much data was lost during the restore from the backup, because people couldn't use it anyway. lol
I also added code to the monitor/restart script that will create a backup of the database every time ieddit.com is restarted. If we, for some reason, have to restore from a backup again in the future, only a couple of posts/comments will be lost.
due to the db errors yesterday, i added code that watched for an error, and then performed a rollback() if the request caused an error. this was the beginning of textageddon.
I woke up today and noticed shit was still breaking, and decided just to add a check to EVERY request that rolled back the request if the request failed due to a db error. This was the equivalent of pouring gasoline onto the fire, will get to why in a bit.
the important part of how comment/post/message/etc are rendered WAS as follows:
for c in comments: c.text = pseudo_markup(c.text)what caused the disaster is that since c is an sqlalchemy object that is stored in the session, and the .text was set as the result of the markup function, if the request errored out, and was rolled back, the c with the NEW c.text would be committed to the db because rollback () would commit the new modified object, unaware/uncaring that it had changed.
this meant that if any text containing html that was escaped, was being commited to the database in escaped form on rollback.
this ALSO meant that the error would NOT occur in the test environment that i had setup, from the DB dump of the live site. since the text in records was already messed up from the very start, it couldn't grow any larger past the 20,000 character length on rollback().
wnewlad also downloaded the db dump, and couldn't replicate what was happening... this is why.
once i FINALLY figured out what was happening, and more importantly WHY it was happening, i changed all comments/text/messages to store the html_escaped text as an attribute named x.new_text instead of x.text, this means on rollback, it will be discarded instead of committed.
i still have to make sure every place which presents htmlescaped user input uses the new attribute name, i pushed out a patch as quickly as possible to get shit running again.
since the DB was now completely fucked up in who knows how many places, I restored from a db dump I made yesterday before going to bed.
and that is the story of textageddon. the site was so fucked up that not much data was lost during the restore from the backup, because people couldn't use it anyway. lol
Alpha btw