Back Seat Programming
One experiment I’m going to undertake is to make this blog somewhat reminiscent of the old .plan files of id fame. I thought that they did an excellent job of sharing various details with the community. I’m not exactly sure how often or how detailed these posts will be, but I know that SL residents will appreciate more visibility into what’s being worked on. Feedback is also appreciated, so let me know if it’s too much data, too little, &c.;
Read on for the info . . .
What we're working on:
Attachments
Attachments cause load on the system in two ways right now. First, they cause many, many updates to your inventory, resulting in excess inventory load (which is already an issue, more in a moment). Second, the create new assets all the time. For those who don't understand our storage infrastructure, most of the data that comes out of your inventory is stored as an "asset." Assets can't be modified, so this allows us to simplify caching and to improve performance. However, that means that if you do want to change an asset, a new asset has to be created. Currently, teleports and logins/logouts cause all of your attachments to create new assets, which is bad. A fix for that is being tested and we'll roll out a preview of it next week so that you can all bang on it. Fixing this should ease our current asset bottlenecks while the longer term asset rework gets done (more on that below).
Inventory
We let you put as much as you want into your inventory, which is pretty cool. Alas, this has also led to inventory bloating and excess loads on our database and login systems. We've narrowed the load down to a few pieces of the inventory and are currently splitting up how we store the inventory. This will allow us to have much better scaling in general, allow many more logins, and reduce the login delay while we ship you your inventory.
Assets
Asset storage is currently centralized. We use read caching to reduce load (hence we can't modify assets) but the write load (mostly due to attachments) is becoming a problem. We are finalizing our distributed asset design this week and will being working on it as soon as the short term attachment fixes are deployed. We are also deploying additional hardware to increase our headroom for the near term. Just about all the problems that SL has experienced over the last month or so spring from asset load, so fixes here will really improve the experience for everyone.
Database Connections
This is esoteric, but it turns out the MySQL has some hard limits on how many other machines can be talking to it at one time. This isn't too hard to fix, but has required some work by the ops folks.
CSR Tools
Second Life is a complicated system, so there are many ways to misplace, lose and/or delete content. We log most of these events but haven't yet written easy-to-use tools for the Customer Service and Support folks. As a result, it takes a lot of time and effort to respond to "I lost my house"-style bug reports. We are going to fix that by making it easier to dig through our logs in an automated way (as He-Man would say, "By the power of Perl . . .") The issue here is that even when objects are "lost" they generally aren't, so better tools (and eventually tools that we can give users access to) will really help this problem.
New Art Tools and Rendering System Refactoring
I can't go into specifics yet -- where would the surprise be then? -- but we have to increase the content team's ability to deploy land and we need to draw the world better and faster. We are moving forward on both those fronts and are very pleased with the early results. Look for some demo movies around Christmas.
UI Refactoring
We have a lot of features and bugs that are compounded by our current UI architecture. We need to fix that, so we have been examining a lot of different options for improving the flexibility and functionality of our UI. As you might expect, this is also related to in-world UIs and text display but don't tell anybody because I don't want to get in trouble for giving away secrets ;-)! I would expect tests and demos of the new technology early next year.
TrackBack
TrackBack URL for this entry:
http://www.typepad.com/t/trackback/1239284
Listed below are links to weblogs that reference Back Seat Programming:
Comments
Ack! You're making SL prettier?
My Geforce can barely handle the load now! :)
Nice to see ya'll are working on the load problems. It's much appreciated. :)
LF
Quick comment on look improvements:
We are making every effort to not increase the minimum spec. However, we do want it to look better for people with higher-end cards.
You're using MySQL?!
Have you looked into PostgreSQL? IME, it handles multiple client/server connections a great deal more efficiently.
Thanks for keeping us up to date Cory, this is something we've been asking for! :D A requested feature about new features, heheh.
These all sound like great and needed fixes/additions!
I think the "lost my house" bug reports would also be cut down more if people had better inventory search features themselves, which it sounds like you guys are working on too.
Are the improved graphics the same thing as when Philip mentioned how lighting was being reworked? Same package I mean.
Also what do you mean by in-world UI's and text display?
Well, I belive I know part of the UI problem-- I tried playing Second Life in Steroscopic vision, and apparently, all of the UI objects are drawn on the opengl level, including the freetype text. --If either lags, all will lag. Though a valid excuse for this is 'transparent windows', I'm sure still you could get off rendering the interface on a diffrent layer. It could improve the framerate vastly (Hint - graphics card Antistropic Filtering / Anti-Aliasing over freetype = death), and also enable Second Life to be played in steroscopic vision, with a readable menu interface.
I belive there would also be a benifit from using a 3d level pointer, as 2d level objects render on the top of the screen and thus in SS mode, the mouse is miles off of where it's actually clicking.
Cory, many many thanks for keeping us so well informed!
Chibi, thanks for clarifying to me why I get chat lag, and pointing out such an obvious solution to getting rid of it... I really hope the programmers at LL use your suggestion on a future release!
Thanks for keeping us in the loop, Cory. I think updates like this are an excellent idea.
In some future update, I'd like to know more about how the development process works at LL. Things like your source control system, IDEs in use, build tools, bug tracking, whether you do nightly builds, that sort of thing.
All those things are exciting fixes, but what about the packetloss meter? Mine is red, jumps from 5-100% packet loss in crowded areas. Wouldn't a sim load really fast with 0% packet loss?
The packet loss sounds like a local issue. I've never seen even 1% packet loss, no matter how heavily loaded the sim. Theoretically, system load on the server side should not generate any packet loss, unless there is a faulty piece of hardware (network controller, router, switch, cabling, etc) or a really poorly-written network controller driver, and those are both things that would manifest themselves for all client connections. If you're showing packet loss, I suggest running some basic tests along the various points between your client and the SL servers, and see what's dropping your packets.
I second Lisse's concern over MySQL! I certainly understand the considerations, in general, that go into designing the infrastructure of a large and hopefully scalable system. I also understand the financial burden that is undertaken with a product such as Oracle. However, experience has also taught me that with very few exceptions, the old adage of "you get what you pay for" is true. I've been in the frustrating situation of having to try to eke the last bit of life out of MySQL in applications that grew beyond projections, and almost universally have had no choice but to migrate the applications to Oracle. In a couple situations the cash outlay required to deploy Oracle was prohibitive, and PostgreSQL showed to be an acceptable stop-gap, at least.
I've never quite understood why people make the MySQL choice. Even something as basic and fundamental as transaction locking was, until very recently, not available in MySQL.
Ohhh i just love back seet programing :p
Doesn't really suprise me that attachments do that. I had a nasty attachment bug where one set overwrote another set in my inventory (i reported it). Instead of having one set of each, i had two sets of one and no set of the other. Oh this will kill my poor 3 year old GF3, guess it's time to buy a new computer.
Chris Altman, if you believe that the constant service wide high level packet loss happening on a regular basis is a 'local issue' you either don't understand what's being talked about or have not experienced Second life in the past few months. The packet loss is simultaneous for everybody, it is a problem either directly on LL's network or via. their choice of data center - it has not really improved since their move of the remainder of server from (and I quote) the 'bad' data center to the 'good data center. LL tried to pretend that it was not their problem for a long time, but it became extremely obvious that it was.
We all experience packet loss at some time or another but for everybody to experience high levels simultaneously in Second Life is not indicative of a local issue, I'm afraid.
Oh boy, RDBMS religion arguments... :)
People love to slam MySQL and say that Postgres is somehow "better." They were doing it in 2001, just as today. Well, let me tell you something I did in 2001. I had to write an app that ate about 2500 rows every 15 minutes. I benchmarked MySQL and Postgres with actual data collected from the devices I was monitoring. Not only was Postgres a lot slower, but my projections indicated that with the then-current growth rate, Postgres would have been totally incapable of keeping up with the rate of inserts going forward (the app SNMP queried carrier-class dialup gear in one data center, and we were going to roll it out in about five more across the country.) Totally unacceptable. At the time, Postgres was notorious for eating tables if it crashed, which MySQL wasn't - another disincentive.
Now, maybe PGSQL is faster today, I don't know... but today, MySQL has full atomicity, subselects, etc... and it has always had really neat features, like AUTO_INCREMENT and LIMIT that are (in my opinion) missing pieces in other database systems... so unless you need a procedural language, what is the point?
Google uses MySQL. What do you guys think about that?
Cory you tease! But I like this blog .plan update thing. Great to see what's on the horizon and what LL's up to.
*grins* Now to get waaay offtopic...
MySQL has always been faster than Postgres. That's because it's barely a database. You get a lot of speed by making shortcuts about data integrity and reliability. (eg. Journaling) Last time I checked, MySQL couldn't even handle tables larger than 4GB. (Dependent on maximum file size)
Google doesn't use MySQL. At least, not for anything important. Google uses a proprietary non-posix distributed file system which is definately not compatible with MySQL or Postgres. (The paper is in the proceedings of SOSP'03 - read it! It's very cool!)
Is the front seat scalable? There's a wealth of development talent out here, and I have no doubt whatsoever that the back seat programmers would love to jump into the front seat with you. :-)
Jokes aside, there is a wealth of manpower available in the open source community. Although SL does employ a few open libraries, the community contribution is not substantive nor world-focussed, and the latter in particular is important if the current snail's pace of progress is to accelerate. It's fair to say that the degree to which one does all development in-house is the degree to which one's development resourcing is non-scalable. That's not an empty adjective: if a competitor with a better development scaling model appears, you're history ... so don't let it happen!
The technical community resource is there to be harnessed. Be creative to find the right development model (it certainly doesn't require opening the whole lot, nor loss of design control), and there will be no stopping SL.
Let us out of the back seat, it's too crowded back here! :-)
Post a comment
ALL OTHER CONTENT MAY ALSO BE PROTECTED BY COPYRIGHT (17 U.S.C.
SECTION 108(a)(3)).—>