A common complaint with relational databases is their lack of flexibility. This ‘lack’ of flexibility has been one of the drivers behind the NoSQL movement. As it turns out, though, there are tools and tricks that can be used with existing RDBMSs that could help you leverage the maturity, tool support, and performance of these systems while still achieving the flexibility you might need with your application. One such trick is the usage of pivot tables.
Continue reading
Funny how a full-time job can pull you away from everything else…
I recently did a major rewrite of Chillax. It’s funny what they say about Common Lisp, how you never finish learning it. Ever since I started using it, I’ve gotten periodic “aha!” moments where something simply clicks. A memorable example was the moment when macros really made sense. The latest one, which came to me while working on Chillax, was the idea of designing your program around protocols.
Continue reading
So the whole reason I even started writing Sheeple was to use it as a persistent object backend for Sykosomatic, that MUD that I’ve been sort-of writing since I started working on Lisp, but is still unfinished.
Sheeple is a very dynamic system, and so when I went looking for a database backend to build this persistence library around, most things fell short of my expectations… and then I found CouchDB.
Continue reading
Adlai said we should make an emergency release of Sheeple, since he added effective reply function caching. I agreed. If you do -any- sort of semi-serious data crunching using Sheeple, stay the hell away from 3.0.2!
It’s taken a while to get this release out — we kept adding stuff on — but it’s finally out. Even though it’s a minor version bump, there’s some important API changes with this new version, including the complete removal of ADD-PROPERTY. Originally, I thought it would be a good idea to force users to add a property manually before being allowed to set a value. Then I realized that the Python community has had this ability for a very long time. The freenode Pythonistas seemed to pretty consistently agree that it had never caused serious problems. Thus, A-P is gone, and (SETF PROPERTY-VALUE) is now used to both add a property, and to set a value on existing ones.
There is also now a manual! A pretty one in both PDF and HTML form, which can be found at our new project page on common-lisp.net. There’s also some mailing lists on that page you might want to sign up for if you feel like tracking Sheeple development.
Things have slowly been ramping up. Ever since I gave that talk at the TC Lispers meeting, a few other people have given Sheeple a whirl. I’m pleased that their experiences have been good, even though Sheeple is still a bit unstable. Of particular note is Patrick Stein’s Wooly, a GL-based GUI toolkit using Sheeple objects.
Development has slowed down a bit since I started working at Clockwork, but I’ve been working with Adlai lately towards a native-Sheeple MOP. So far, the basics of object creation have been taken care of, and a property MOP should be up and running soon. We’ve also decided to rename persistent-sheeple, but that will get announced once we have enough of a MOP to hook up our objects to CouchDB.
Posted in announcements
Tagged lisp, metaprogramming, object-oriented programming, release, sheeple
Leave a comment
Sheeple 3.0 is finally out!
After months in development, it’s finally done. A major rewrite of the entire Sheeple system, this new version boasts a completely new, cleaned up, and serious’d interface, reworked, lightweight objects, and efficient property access (right now, matching CCL’s slot-value performance.) You can download a tarball here, or use asdf-install to grab the latest version.
I’m really excited about this release. I would write more about it and all the goodies it has, but I have a presentation to finish. Expect more in a later post.
Okay, so I haven’t actually posted anything about Sheeple in a while.
Sheeple took a break for a few weeks, but development is up to full steam again. There’s some shiny new special sauce going on under the hood now, with some -very- promising early results.
See this paste for some picobenchmarks on various implementation. Notice: direct-property access is now as fast as slot-value on CCL. :)
Unfortunately, the rest of Sheeple doesn’t perform that well quite yet. The new fancy secret sauce allows for a lot of optimizations that weren’t done before. Once this code is stabilized and tagged as 3.0(!), I’m bringing back a bunch of different caching schemes Sheeple was using for dispatch. Tonight’s benchmarks were promising — I think Sheeple may actually be able to perform better and faster than optimized CLOS implementations such as CCL’s and SBCL’s.
We’ll see. We’ll see…
For the longest time, I wanted to start learning about issues related to concurrency, and how to handle them in Lisp. I kept reading around to learn about different approaches to writing concurrent code, and studied up on things like Software Transactional Memory, futures/promises, and Erlang-style actors. After boggling to understand all these different approaches, I was left with a bit of disappointment: None of them seemed to be the right combination of generalized+clean+easy.
STM caught my eye for a while, but it still has its issues, and I’m still not sure how I feel about the whole “thrash until you can agree on something” issue — it seems to me like it’s just the same as wrapping code in (with-lock-held …). Futures/promises looked easy enough for simple one-shot tasks, but that’s not necessarily what you want to do when you write heavily-parallel code: Sometimes you want threads constantly yielding values. Erlang-style actors are, of course, one of the big success stories when it comes to people trying to write heavily-parallel code. It just seems so damn ugly — and it -is- nice to be able to share data sometimes.
Then, magic happened: I came across something called Communicating Sequential Processes, through Rob Pike’s Google Tech Talk on Newsqueak. It was really amazing, it seemed to be just what I needed: a relatively low-level synchronisation mechanism that works naturally with the concept of multiple parallel threads/processes. I’m totally sold on Rob’s point of view here: don’t try to pretend you’re not parallelizing code (hear that, STM? That’s right). Instead, make the parallelization part of your interface. Deadlocks are a bug — find them.
Continue reading
Posted in random
Tagged chanl, communicating sequential processes, concurrency, lisp, threads
Leave a comment
I’ve been working on rewriting Sheeple these past couple of days. The new version will be a major bottom-up redesign, mainly to the lowlevel stuff. The early results without too-crazy optimization are already showing up, and it’s quite nice.
These numbers are on Clozure CL x86 Linux:
SHEEPLE> (time (loop repeat 100000 do (allocate-std-sheep)))
(LOOP REPEAT 100000 DO (ALLOCATE-STD-SHEEP)) took 119 milliseconds (0.119 seconds) to run
with 2 available CPU cores.
During that period, 106 milliseconds (0.106 seconds) were spent in user mode
3 milliseconds (0.003 seconds) were spent in system mode
50 milliseconds (0.050 seconds) was spent in GC.
4,000,000 bytes of memory allocated.
19 minor page faults, 0 major page faults, 0 swaps.
And a single sheep object:
SHEEPLE> (time (allocate-std-sheep))
(ALLOCATE-STD-SHEEP) took 0 milliseconds (0.000 seconds) to run
with 2 available CPU cores.
During that period, 0 milliseconds (0.000 seconds) were spent in user mode
0 milliseconds (0.000 seconds) were spent in system mode
40 bytes of memory allocated.
Now, that’s still a bit heavy compared to CLOS (which only allocates 24 bytes for a raw instance), and to be honest, I’m not aiming at beating CLOS. It’s like a class-based system trying to beat out structs. On the other hand, it’s actually pretty nice to shrink everything down to more usable levels.
The current Sheeple release, for example, the one built on top of CLOS, is quite a monster:
SHEEPLE> (time (loop repeat 100000 do (allocate-sheep)))
(LOOP REPEAT 100000 DO (ALLOCATE-SHEEP)) took 11,273 milliseconds (11.273 seconds) to run
with 2 available CPU cores.
During that period, 10,716 milliseconds (10.716 seconds) were spent in user mode
204 milliseconds (0.204 seconds) were spent in system mode
3,249 milliseconds (3.249 seconds) was spent in GC.
244,000,000 bytes of memory allocated.
9,419 minor page faults, 0 major page faults, 0 swaps.
So yeah, this is a huge improvement.
Keep in mind that all of these numbers don’t yet take into account what the full cost of a raw (clone) will be, although it won’t be much higher than what it is right now (I believe it will only be two extra words, because an item will be added to the parents list).
It’s a bit fun to mess around with stuff so carefully at a lower level…
Posted in random
Leave a comment
There’s been some work done on Sheeple’s MOP, and I figured I’d write a bit about what got done today…
The original reason for writing Sheeple, like I said before, was to use it with Sykosomatic (which I still haven’t written an entry for, meh). That said, Sykosomatic’s object system needs to be fully persistent. Sheeple itself is not persistent — it works like a standard run-time object system.
Because I’m a masochist and a nerd, I figured I’d simply write Sheeple as the regular object system that it is, then “just” add a Metaobject Protocol like CLOS’ once the base language was set. Ho boy, did I ask for trouble. MOPs are a royal pain in the ass to do properly, and they’re a bit mind-bending. Regardless, Sheeple has been slowly getting MOP features over time. The neat part? The MOP itself is written in CLOS (as are a lot of Sheeple’s internals).
What I managed to get done today is the protocol for manipulating properties and property behavior through property metaobjects.
Continue reading