What is grep and How do you use it?

That's my random thought today. Given the success I had last week with find and xargs I am inspired to understand the most common unix utility I hear about. The mighty grep.

What is grep?

The technical answer is ok but leaves you guessing. Gnu says

grep prints lines that contain a match for a pattern.

in their grep section

A better definition comes from HowTo: Use grep Command In Linux / UNIX – Examples

The grep command is used to search text or searches the given file for lines containing a match to the given strings or words.

How do you use it?

HowTo: Use grep Command In Linux / UNIX – Examples provides some nice straightforward examples. Though I couldn't get the search working in a folder, just on a file.

grep php /mod/quiz 

returned no such file or directory. It will be user error but it's what the tutorials show.

Though I am using GIT bash on windows 7 MINGW32. That could be the problem.

Turns out I need to prefix the command with GIT e.g.

git grep php mod/quiz

this worked fine but returns too much. So I then tried a search with a smaller file base

git grep link question/type/calculated/lang 

This returned a lot less. I also had to remember to type :q to quit the command in git bash. Note that I don't prefix the path with / this returns and error. ./ works but isn't needed. 

Atleast now I know the basics. I'll find a use for it soon I expect :-)

Can you run phpunit tests on entire folders?

I'll warn you up front. This is a post about code and programming. If you've already fallen asleep it won't get any better :-)

My day job is about code and I see the purpose of it all as finding faster ways to do something than before.

Today we have quality control duties. We upgraded to the latest version of Moodle (2.6) and so we're checking all our plugins work. This means running a bunch of boring tests. Some automatic, some manual. Most not as automatic as I would like.

So of course this is a chance to learn something.

Moodle uses PHPunit and you run their unit tests through the command line. The advantage is that you can then use the power of the command line to automate things or at least that's the theory.

In this case we use Git Bash, not the windows command line itself. This gives us unix like functionality and a chance to learn some unix. Useful for a guy like myself who got trapped in the Microsoft castle early on in my career. I've been struggling to get out ever since.

I have the task of checking 12 question types, 3 reports and 2 other features. My first thought? How do I automate thee.

Individual tests

I started running each individual test with commands like

vendor/bin/phpunit qtype_ddimageortext_walkthrough_test question/type/ddimageortext/tests/walkthrough_test.php

learn more at phpunit: the command line test runner
Then I realised I needed to do this for each test file of each feature. That's a lot of files

Individual Folders

I asked around and came up with
find question/type/ddimageortext/tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit

This taught me some Unix. I didn't realise that in Unix you can query the file system much like you query a database through sql. Fascinating.

What this sentence does is first find files for a given query then use xargs to execute the phpunit tests with the required parameters

Find

The find command is everything before the |

find question/type/ddimageortext/tests/*_test.php -type f -print

You break it down as follows

• find : Find items given relevant parameters
• question/type/ddimageortext/tests/*_test.php:  all files in the given path matching the *_test.php expression
• -type f : match files. For folders use d
• -print : Print results to the command line. Otherwise you don't see anything.

xargs

The xargs command is everything after the |

xargs -n1 vendor/bin/phpunit

You break it down as follows

• xargs : create a series of command line statement items given a utility and argument
• argument -n1 take the first item passed from find as an argument to follow the phpunit statement
• utility vendor/bin/phpunit phpunit is a utility xargs can call

That broke my head a little at first but now I have described it I finally feel I actually understand it :-)

Which test has been run

This was working really well. I could see all the tests in one folder had passed though I realised I didn't know which test file was being called each time. There was nothing to tell me the file name or relevant classes. Not critical but I could see the potential to search all question types and thus multiple folders. At that scale I need to easily identify which files have problems.

enter phpunit --debug
The --debug switch is the only option I could find to print out anything related to the test file being processed. It prints too much info, each individual test that is run but it's better than nothing. So I went with it.

That built the command up to

find question/type/ddmarker/tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit --debug

All folders in question/type

Then of course I saw the potential of just running all the tests in the question/type folder because I'm going to test most of them anyway and it's quicker than checking each individual folder.

So I try:

find question/type/*tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit --debug

and find two problems.

1. There is too much information flashing by. I need this stored to a file
2. The run aborts when it reaches the stack folder. Ignoring the tests after the stack folder

Save to file

This one is straightforward just add

> unit_tests.txt

to send the results to a file in the location your shell is currently at.

Exclude a folder

I tried various options to get round the stack errors including:

• phpunit::filter patterns to avoid tests that errored and stopped the run.
• find only specific folders

The first didn't work. The second just seemed more complicated than my final solution.

So I ended up telling the find command to ignore the stack folder. It took a little doing but once I got my head around how find works it became straightforward. 

I just needed to tell find to exclude anything with a path of stack. I did this using -not -path "*stack*"

Ending up with:
find question/type/*/tests/*_test.php -type f -not -path "*stack*" -print | xargs -n1 vendor/bin/phpunit --debug > unit_tests.txt

I just ran the command and it worked perfectly. I now have a file listing the results of all the unit tests in the question/types folder. I found two warnings of incomplete tests but nothing else. Result!!!

These are some of the references I used from the search find all directories except

• exclude-directory-from-find-command
• Running find in all directories except one
• unix find command examples

Of course now it seems simple. The point is that I learnt a lot about unix. I now understand two unix utilities and that the point of unix is that it is

a bunch of utilities that you can link together in various ways like lego to achieve any end you want. 

This is quite literally a small step for my coding but a huge leap for my understanding of operating systems and why unix is the father of all we have today. I now understand why everyone else reveres it so much.

Templates

This would all seem like a lot of work for something you do rarely. That's what everyone tells me when I share this. You see I log all this and keep it for later. Last time I built a nice template for testing which I just brought out again this time and copied as a basis to begin with. I have the same work to do plus a few more features to test.

Now I have added much more to the template so I can get the job done to the same standard next time but faster. I've always found this iterative approach a great way to learn and also improve so I'm sticking with it. The only difference is that I'm now sharing it more openly. 

Spongelab: Build a body

Just recording the sponge lab game Build a body that I just found. Click on the launch interactive button to play the game.

It's like the google virtual body but also part of a much wider assortment of learning tools on spongelab. Very interesting to find.

Spongelab calls itself

The best in digital learning.

and claims to be

A global science community 

So of course I want to explore this site over time. I'm starting  with their Build a body game.

I could pick apart the interface and interactivity but I think that's unfair. I find it quite simple and straight forward. I didn't need any instruction. It just worked. You place parts of the body where they're supposed to go and learn about them as you go.

What more could you ask?

In truth simplicity is great for learning. This tool doesn't try to be more that it needs to be to demonstrate the concepts. So I like it. Of course I would suggest it be made in either HTML 5 or unity for cross platform support but that's just a future feature. The concept it what is important and it's done well in my eyes.

So I'm looking forward to looking at the other resources sponge lab offer.

Is Vanilla JS ready yet?

There are so many javascript libraries that I can't be bothered to learn a new one without good reason.

This tweet

♇4n3 ‏@Pl4n3 TIL vanilla.js mind=blown. :)

led down an interesting path. I visited vanilla-js.com hoping for some kind of demo. Nothing there. Then this line caught my eye

Vanilla JS is so popular that browsers have been automatically loading it for over a decade.

hmm.  Now that is interesting. Checking back to the tweet p4n3 is a 3d guru I have respect for. If he is using it I might be interested. Looking through the speed comparison tables was worthwhile.

I was hoping that javascript itself has matured to the point that the need for libraries is reduced or non existent. Thankfully that is what this site and the tweet is about. Worrying less about what library you use and more about solving problems that matter.

The link to Vanilla JS documentation points simply to the Mozilla Javascript page.

A few reviews make interesting reading when comparing native js with libraries

• Ditching jQuery for Vanilla JS | Go Make Things.
• Thinking Inside The Box With Vanilla JavaScript | Smashing Coding
• youmightnotneedjquery.com a site dedicated to vanilla js (thanks to @tim_hunt)

Each highlights that native js can now replace the need for several aspects of libraries. This is good news. I'll keep an eye out for places I can use it

For me though I'm going to wait. Simply because it's an evolving area. A few more years and I might just be able to write pure JS. Right now though I have to figure out which works every where and which doesn't in some random case. I have a life too so I'll let others tell me when it's all ready. 

Glad to hear JS is well into its journey to maturation though. 

Pairing vs. Code Review: Comparing Developer Cultures

Inspired by a great post Pairing vs. Code Review: Comparing Developer Cultures  by Paul Hinze of So Create I wanted to keep this for reference. We do code review in the VLE team at the OU but not pair programming. I can see its value but I am much more in favour of variety of approaches then only using one. You might favour one over the other but in most instances it's more important to have a few options and you git the approach to the solution.

That said I've never done pair programming. I'd like to try it just to see what it's like but I expect to get nervous having someone stare over my shoulder like a back seat driver. I've done something close like I think we all have. That is ask some one for advice and we then try and solve the problem together. You start writing code and essentially programme as a pair. The difference is that you are still the sole responsible person for the task. It didn't start out as a joint task.

So reading some one elses experience seems useful. One particular aspect I'm looking at is the economics involved. My degree included management and it gave me a passion to understand the long term value of things. Pair programming seems great for quality but you're paying two people to complete a task. So it's a bigger investment than paying one. Is the return worth it? That's a very open question that's more about interpretation and personal choice than fact.

Here I summarise what I have taken away from Pauls account so I have a good record to refer to quickly.

Prerequisites for success

There are a few nonnegotiables that are common across both of these paradigms.

• Solid continuous integration - builds that run on every commit
• Talented core developers - these folks helps promote quality and drive the architecture of the codebase
• Agreement on the importance of code quality - the team and the overall company both recognize the value of maintaining a high-quality codebase
• Iterative self-organization - the team is willing to collectively evaluate their process and course-correct on a regular basis

In my experience these things are all necessary no matter what process you decide to use.

Advantages of pairing:

some of the advantages Paul has found with pair programming

• high bandwidth communication between pairs
• You can train junior developers easily by giving them an experienced pair. 
• Core developers can spread best practices and knowledge silos across the team quickly. 
• New tools and techniques are shared naturally across the team. 
• Everybody gets better together.

Dis Advantages of pairing:

The disadvantages paul has found are:

• Pairing is sensitive to team balance: Pairing experienced people with junior and over stretch the experienced staff, It can also break down knowledge silos but also make it hard to find good pairs. One knows the area well, the other doesn't and slows the team down.
• Pairing culture can breed monoculture
• can make it tough to hire people. Not everyone likes constant pairing and working closely with the same person all the time. It can be harder to introduce different tools and approaches.
• Pairing does not lend itself to problems requiring in depth thought like system design and architecture: Particularly an experienced programmer paired with a junior. Teaching while solving a difficult problem is a particularly tough challenge.
• Sometimes you end up needing code review anyways

Advantages of code review

Paul feared the transition to code review but actually enjoyed the experience. Some benefits were:

• the positive motivating pressure to perform well is still there.
• In contrast to pairing, there are no barriers to taking on deep thought problems.
• lends itself to a wider pool of personality types that can be effective
• Code reviews are asynchronous
• forces you to think about your values more

Dis Advantages of code review

• being alone: less camraderie
• more emphasis on your own self control. You can be lazy.
• reviews and associated discussion can stack up
• conversation happens after something is built. 

What's best?

My view is that they are both useful. What's best depends on the problem at hand and environment. Paul agrees. He finds both work well. His closing note emphasises the focus on quality not workflow.

So if you don’t have one already, find a team that cares about code and about process and you can try to solve some of these problems together.

Git: Fixing “Your local changes to the following files would be overwritten by checkout”

Git is quite a learning curve. I get the feeling I can now call myself experienced because I'm starting to search for quite indepth solutions.

Today was how to get past the error message “Your local changes to the following files would be overwritten by checkout” when trying to switch branch.

The fix was actually straight forward in this case.

git checkout filename

 The background was more interesting.

• git status showed no differences
• gitk also showed no changes
• this is because I am using git --assume-unchanged a lot more for key reasons

It's the --assume-unchanged aspect that complicates things. I'm not happy to have to do it but given the work I have to do and the rules that go with it it seems the best option. That is a discussion for another time. For now this answer helped me so I hope it helps you. 

I'll now go into a little detail why the fix wasn't as obvious as it seems by showing the symptoms you get when you use --assume-unchanged

The extra detail I got from the error message was 

Please, commit your changes or stash them before you can switch branches.

It is this wording that introduced the complexity. Git was telling me a file was different when I called checkout but not when I called status ot gitk. Confusing 

My search didn't give an answer straight away. Had to do a little hunting and of course the answer was nice and simple. 

I tried:

• git stash:  no joy because no differences were found on the branch
• git stash save --keep-index same as git stash. No difference to stash.

Gene Experiment

Here is another approach to exploring horizontal gene transfer

If you have created some simple bacteria in a virtual world like Unity3d. They don't have to be complex, just cells or even just spheres. Get them to move around a petri dish. They could have a nucleus and inside that they have genes as scripts. Each script has an effect like:

• glow in the dark
• turn red, green or blue
• get longer or shorter.

You can then selectively knock out these genes and see the effects over time on them and their offspring. You can mate the bacteria to see which genes are expressed. Just an experiment on genetics. Something to share and start the ball rolling. Save everything in json so anyone can use it.

I expect this has already been done elsewhere. If so great but most examples aren't designed for others to build on. That's the idea here.

• Unity is free to use
• the code required is relatively simple
• sharing the application and source code with others is relatively easy. 

Not only do you demonstrate the principles of genetics and gene transfer but you teach others that they can practice the principles of biology directly through digital research.

That is what really excites me about knowing how to program. I don't have to wait for my cell culture to grow. The only limit to what I can learn is my imagination. I can literally create my own laboratory and experiments whenever I wish. Fun :-)

The Plague and the benefits of Horizontal Gene Transfer

How do you express the concept of Horizontal Gene Transfer in a way most people could understand and also see where it could be useful?

In fact I have several related concepts and questions I think I can express in one idea:

• What is Horizontal Gene Transfer
• Is  horizontal gene transfer natures software update?
• Can viruses can do good?
• Can programming teach you about biology?
• Is biology built into programming?

These are the challengea I've been considering for a few years. One idea I like is creating an experiment to demonstrate the concept in a historical context. Once I understood Horizontal Gene Transfer I quickly started to wonder if this would be a simple answer to why certain people are immune to pandemics like the plague while others aren't.

The point is that anything that would help you survive the plague would be selected for in an evolutionary fashion. Even in the most deadly plagues and pandemics there always people who are immune and survive. It makes a lot of sense that if this immunity were the result of a viral infection then it could spread.

As long as the virus doesn't harm the individual enough that they don't survive then the virus would actually help people live when those around them would die. This would become a symbiotic relationship as long as the plague lasts. It would also encourage the transmission of the virus. Those who are immune would obviously become more common relative to those who are not immune. Stories of miraculous recoveries by being touched or in the presence of certain immune individuals would be abound.

Viruses survive because they can change the genetic code of the hosts they infect

This may sound fantastical but the concept is already in place and called active immunity. The individual themselves would produce the immunity. A related example is Sickle cell anaemia. It is dangerous and uncommon in normal populations but more common in places where malaria is prevent.

Sickle Cell Anaemia protects against malaria. So while normally it would be a bad condition to have but in an environment where Malaria is common is would be quite a benefit. So it demonstrates the principle of an affliction being an advantage however Sickle Cell is an inherited genetic disease.

All we are really talking about here is for nature to come up with a way of being able to transfer this condition to others. That's actually what viruses do. In fact it's how they live. Viruses survive because they can change the genetic code of the hosts they infect. Otherwise they wouldn't be viruses.

So that's the basic theory. A virus uses horizontal gene transfer and could infact do good instead of harm. Giving a species selective advantage. How to demonstrate this though, that's a fun challenge.

We have always known more about evolution and biology than we think. We just haven't appreciated enough different viewpoints. 

So the context is the plague. I'm obviously thinking of the great plague that affected Europe in the middle ages and came to England because I'm English. You pick your own similar story that interests you. The concept is most important.

Working in software is for me an advantage. I practice horizontal gene transfer on a daily basis so the concept and its application becomes quite straightforward once I'm in the right mindset.

The story I want to follow is of a cell or cells being attacked by plague bacteria. You must introduce a virus containing code to protect against the bacteria. Essentially an update to the cell code base. Now the exploit on the code has been fixed.

This explains in one example how a virus could protect us against infection and thus be beneficial. Not just harmful. And how our genetic code can be constantly updated through exposure to the world around us. Just like our devices and apps are constantly being updated.

I described how to translate this into software so this article is about putting context and story behind the techniques. Why you would bother to create this app and why Unity interests me so much as a development platform. You could do this through html 5 but I think it would be easier and have more impact through Unity.

Overall This demonstrates neatly my view that we have always known more about evolution and biology than we think. We just haven't appreciated enough different viewpoints.

Build a Virtual Body: Bacteria communicate with their hosts

This article Communication between bacteria and their hosts is the kind of thing I look for to support my dream of a virtual body. My interpretation is essentially simple.

I consider bacteria and viruses to be to the human body like plugins are to software. The body is an entire organism but in fact it's a symbiosis of multiple organisms that evolved together. These organisms are so interlinked they couldn't exist separately.

Normal evolution means that the body must continually keep up with the changing environment in which it lives. Bacteria and Viruses are simply natural means to facilitate these changes. Horizontal gene transfer is part of this mechanism and there is plenty of evidence that the organisms we host talk to each other. This is just the same as software and operating systems. They all talk to each other. Lots of tiny applications and scripts slowly get incorporated into software and ultimately built into operating systems. 

I simply see the same pattern happening in biology. In fact I feel that we are just seeing the normal processes of evolution occurring in digital fields as we have seen in the biological ecosystems. 

Updating cells through Horizontal Gene transfer: An idea for a game

My interest in biology runs so deep I enjoy coming up with my own theories. If you've read my blog you'll find many. Of course what is most exciting is when I find that a theory I have developed from my own experience and logic turns out to be real. The process of horizontal gene transfer is just that and is becoming more widely understood as a far greater force for evolution than sex.

 “The human genome includes sequences derived from viruses.”  Page 135 of The mathematics of life

Not only did an understanding of software development and thus code help me understand this most basic aspect of allowing a cell to upgrade its DNA but I feel it has great potential to express science concepts particularly biological ones.

That's the reason I have been exploring HTML 5 and 3D technologies including unity for the last few years.
From what I have learnt I think I can now show the concept of horizontal gene transfer in ways it hasn't been shown before.

The challenge I face is one of resources, time and money being a big issue and skills. I don't know the 3D virtual worlds arena well enough yet. So I thought I would simply describe the basic specification for the small game or app I would like to develop. If you would like to help in some way please let me know. I have so many of these ideas that aren't really that hard to put together but other constraints get in my way. It's the bain of our lives isn't it. Hopefully this is of use to some one.

Plugins

The basic concept is that human code like PHP, Javascript, C etc already has many of the characteristics required to demonstrate Horizontal Code Transfer. If you consider the concept that many pieces of software like moodle, wordpress or web browsers allow plugins. The plugins work because they are designed for the target code base. They do what is necessary in the target code base to achieve their ends. That is pretty much how viruses work. They are targeted for a particular species and have a specific mechanism by which they adjust the target DNA code.

Simplicity

To keep things simple lets just focus on a cell, ignore the other cell components for now just have a nucelus and a virus. The virus contains the new code the nucleus. contains the target code base. The virus gets into the nucleus and delivers its code.  The concept is simply that the new code is a script/plugin written for the target code base.

In Unity3d this is actually quite a simple game to create. I've done most of the work already but the time to put it all together hasn't arrived. If you want to know more just ask. I want to be able to teach this to my son when he is older because it is such a fundamental way that our bodies work. Now that I understand it I can't tell you how useful it has become.

An experiment

I would create an experiment. Have a cell with a nucleus with a script attached that controls the nucleus and cell. A virus gets through. It has a script attached. It attaches it to the nucleus. The new script adjusts the existing configuration of the cell. Using the existing nucleus as a code base. It then creates more of itself.

It's an experiment to show It's possible in code and the mechanisms behind horizontal gene transfer. An added bonus is that a knowledge of software creation is can help understand it. I would then look at ways in which this could benefit a cell. Much like an update for your favourite program.

Modelling evolution

Using the cell model I was building. I would start to assemble the cell through evolution like this. I would then make a game around it. Maybe the player manoeuvre each element into place, or players build up their own cell and lots of viruses try to attach. The player has to get the useful viruses in place while avoiding the bad viruses .

This could be an open source game like a cell manager. You play the cell security. Choosing which viruses get through.

What can you do with chrome://chrome-urls/

So my last blog shared a couple of useful pages for managing chrome plugins and crashes. I just published the post and then found something even better. A page in chrome that lists all the pages that might be potentially useful. Just type chrome://chrome-urls/ into your chrome address bar. It lists every useful behind the scenes page. Just click which ever you want to find out what it does.

Heck I even tried it on Android and it works. Yep. Very geeky but I'm going to have to store that one in my mind some where. So if this nugget helps you then please let me know. How did I find it? I just started typing chrome into the address bar to see what would come up. It did the rest. How useful it that.

Bacteria ‘hijack’ human immune system

Just logging a fascinating new insightfrom the Royal Society that Bacteria 'hijack' the human immune system. It's no real surprise to me. I've talked about how we are made from bacteria and this can both benefit and harm us

The lesson from this article is that bacteria are continually evolving. Most live within us and help us. But we are to them just like the earth is to us. We are they environment in which they live. As well adjust they adjust and as our immune system seeks out harmful things it looks for bacteria like any other potentially harmful organism.

Without going into detail this understanding can have implications for cancer, recovery, allergies, activity, diet

Learning is innate. How the changing world can help education

The evolution of learning is a fascinating topic. Learning is my strength because I was brought up to think that learning is innate and every challenge is surmountable. In fact learning is never really changing, only the tools and processes we use to learn will change. The education industry has been struggling to figure out how these new tools can help people learn.

I've never thought it's that difficult because learning is about the human process, the tools are just facilitators. I've been waiting for years for my employer the Open University to realise this. Listening to a 2012 HEA conference keynote by Martin Bean my CEO/Vice Chancellor excites me because he seems to get it. So much of what he says in this keynote reflects what I think. That is why I really like what is happening under his watch at the OU. I simply feel that all the new things we can do these days are simply what we have been waiting for.

The most sophisticated technology in the learning process is always the learner not the tool used to teach them. Learning is really just a case of getting information into a persons head and body and then helping them figure out how to use this information to get something done. You have learnt something when you overcome a challenge that you previously couldn't. That is how I see it.

Humans have an incredible capacity to consume information and get it into their brains. We have senses including eyes, ears and touch which we can use all at once. The quantity and quality of data these senses can consume is terrific. Newer tools are just making it easier to make full use of these senses. Humans are also able to combine their senses to achieve secondary or tertiary senses like proprioception and kinaesthetic awareness. It is now getting possible to speak to these senses through devices like the kinect.

The point here is that people will pay for the ability to overcome a challenge that matters to them. If you can make this happen faster and more easily then people will pay more. So, learning isn't really changing because humans aren't changing. They're just doing what they always did but with better tools and more support. So standards have risen, content is now cheap and the market is exponentially larger. Surely that is a huge opportunity.

What is needed is simply going back to the essence of learning and re interpreting existing content, tools and assessment methods in light of what is possible today. Surely we can make the learning process the innate, natural and easy process humans are designed for?

Many thanks to Jonathan Vernon (@mymindbursts) who inspired this post by sharing Are we at the Napster moment in HIgher Education. A thought provoking article.

Spore: Why it didn't work

A dream of mine is to create a game or piece of software incorporating evolution and cellular development. So I was very excited to read Spore: My view of the Elephant written by someone who actually worked on the game Spore. It is a highly worthwhile read. Capturing both the complexity of the task and also the lessons learned that doomed the project to failure.

Let me know what you think.

David Attenborough. Rise of the animals and the rise of simulation

I have always loved simulations for their ability to put concepts into the context that makes most sense. I am developing such respect for David Attenborough as I see his ability to take this relatively niche medium and bring it to the mainstream. Using it with his usual attention to detail.

The show fascinates me because it shows the progress of evolution and because I just love anything to do with nature. What makes it worth writing about is the use of 3d and simulation technology to immerse the viewer in the natural world as it was. This isn't new but the level of quality with which it was done shows the evolution of this tool and of the people using it. It shows how 3d, simulations and virtual worlds are being used to express biology in ever more realistic ways.

I prefer it when the story and concept is foremost and the tools used to convey the story disappear into the background. Something they have achieved in this series. A favourite scene of mine depicts a Tiktaalik vertebrate that is extinct as though it were alive and moving from water to land. The treat is that the whole landscape is so realistic despite being a virtual recreation. It literally would not be possible to re create this in real life. It is only possible in the virtual world yet, at least on screen, it appeared anything but virtual. 

My interest in 3d is also because I am learning the skills of producing virtual worlds and simulations using tools such as Unity3d. It is pioneers like David Attenborough who are my teachers because they are using these tools in the way I hope to. Their efforts make it easier for me to show others the value that simulation and similar tools offer.

Just like the simulation used in the hidden life of the cell it's such a rich way of expressing our knowledge.

Brain Facts 09: Aging

Just listening to episode 9 of the Brain facts podcast. It is like going back to my psychology A level. Something I really enjoyed. The beauty is simply in understanding the body as a whole. So many know a lot about the body but little about the mind. This is the same as knowing all about a computer but not the operating system it runs or a car but not the fuel and engine it uses. The knowledge is incomplete.

This particular topic on aging reminds us that the brain is there to be used as we age just as when we are young. There is no evidence that any decline is inevitable it is just that that is what normally happens. So all you really have to do is choose to use your brain as much when you age as when you are young. In fact, as far as I can learn from my seniors it is the experience you possess when you are older that enables you to out compete the younger fitter individuals. My plan is to keep my fitness while gaining the experience so I can have the best of both worlds when I am older.

Listening to this review reminds me of what I learnt so long ago. That is a completely achievable dream.

Could education learn from the Google docs business model?

Building on a blog post ive written discussing the notion that flow of resources is key to life. A colleague helped me understand how the same can be said for business models and can explain how to make money in this new connected age. How services like google docs could make good money and education could scale itself. 

In this case we're talking about how google can build a business by selling their tools. Gmail and gdocs are just their tools. Gdocs doesn't contain adverts so where's the business model?

To start with you can by extra space when youve used up the free allowances But that doesn't explain how they can stay competitive. We had to look a little deeper to realise that cloud providers can do things that local organisations and individuals find difficult. Real economies and practices of scale. 

We realised its actually just like the economy being built from banking. If every person withdrew all their cash the banks couldn't actually give it back. They only hold what's necessary to give people what they need day to day. They resell the money they hold and that's how they make money.

This is because it's cheaper for restaurants and car manufacturers to focus on what they do well and pay someone to look after their wealth. It's nore profitable for banks to see their cash as stock and resell it than to lock it up. That's just how the economy and most mature industries work and thrive.

So that's how google make a profit in software and tools. It's too expensive and a waste of effort for every company or individual to build and maintain their own web based office suite. Better to use one from a brand you trust. To make it profitable Google will not have the capacity for every person to use all their allowances at once. It's too expensive to actually buy and maintain the hardware and resources to support this. And also no one uses all their allowance all the time. It also doesn't matter if each persons data is in one place or distributed all over the place. 

All that really matters is the current demand. As long as Google can pull it all together and present it in the way the customer wants and expects they're happy.

So Google have so many customers they'll work on trends. They see documents and tools as data and customer traffic as data traffic and requests for data. Each day an average amount of new data storage is required. It will vary over time but this can be predicted with certain contingencies in place to cater for unpredicted events.

Once you master this you can build a reputation for global reliability. You always "just work". The price you can charge and the loyalty you gain for such reliability and dependability is huge. And this approach to it is cost effective and thus profitable. Thus it's a business model.

That is why I want to bring this to education. This is enterprise structure with enterprise profitability with an enterprise approach. It's what education needs. A platform and infrastructure that's built to scale to modern standards using time honoured approaches of achieving scale. Stand back and see things for what they are. Honesty, no warts. Go back to what you're good at and what people want. Reapply all you know but with a modern focus.

Developing for Google apps. What do you need?

Image via CrunchBaseJust having a bit of a Google day to day. Started talking to someone about Google Web Toolkit and App engine. From that I realised I hadn't looked into how well it scales and how robust it is. I've just been assuming it will grow in time and do everything :-)
Had a quick look at the Google Marketplace and confirmed that GWT apps could be placed there. The marketplace is available for those using Google Apps for domains where you get google docs, mail, calendar and sites designed specifically for business use.
From there I checked out the options for building for Google Apps. It's nice to see you have a frew options from simple javascript applications right up to copmlex java and python based fully hosted applications. Turns out it's all about apis and I promptly found a link to all google apis on code.google.com
Since I'm becoming a big GWT and appengine  fan I'm sure these links are going to become very useful very soon.

On the origin of life

Image by pericomart(pericomart.blogspot.com) via FlickrI've had this idea floating in my head for weeks now. So I'm finally getting round to putting down into words. I'm excited to finally express it but I hope I do it justice. Here goes.

We're all aware of the theory of evolution. It makes perfect sense to me how pressure for resources has lead to so much diversity in life over such a long time. However evolution does not account or even attempt to explain how life began, the origin of life, at all. It merely explains why change is the norm and that life is about adapting to change. 

What I have come to believe is that life is basically about the exchange of energy. With this exchange we can fuel all sorts of processes that have built up over time to the natural world we are used to. It also powers the amazing senses with which we get so much pleasure.However the exchange of energy must lead to processes which ultimately bring more energy into the organism so that more energy can be exchanged and the cost of converting the energy into another form must be less than the energy gained from the exchange.

It's exactly the same as the environmental problems we keep hearing about. Right now it takes more energy to get oil and coal etc out of the ground and into our cars as petrol and homes as electricity than the amount of energy actually produced. So it's taking from the environment without giving anything back to balance things out. So it's unsustainable. This certainly happened in the early days of this planet. The atmosphere was very different and the organisms that found life there eventually killed themselves off because they changed the atmosphere. So what was good for us was bad for them but they only have themselves to blame.

With this example, it strikes me that life in itself appears constant. Organisms live and die but there always seems to be something that's alive at any given point in time. This may not have always been the case but it seems so now. So the organisms that have life are constantly changing but there is always something that has life and often the death or demise of one species is required for another to live.

Studying biology I am always reminded of feedback loops. That insulin and glucagon work together to regulate sugar supply in our blood and cells. If there's too much sugar in the blood your body releases more insulin, if there's too little it releases more glucagon. Adult onset diabetes occurs when this natural feedback loop breaks down because the bodies cells no longer respond to insulin as they used to. The feedback loop that corrects this naturally is that of activity (exercise). The extra demand for sugar in the muscles forces the body to become more sensitive to insulin again correcting the diabetes and bringing your body back into balance. 

These feedback loops are so common that it's easy to feel that they are the basis of human life and most of nature. Each organism is simply a collection of feedback loops that all, in theory, balance each other to maintain life. Death often occurs when one or several of these loops is out of balance for a long time. 

This thinking lead me to believe that life itself really could be one long feedback loop. For example if you don't reproduce, then your gene pool could die. With so many humans on the planet it won't have such an impact now but back when there were only a few humans on the planet this really would have mattered. But essentially for a species to survive then it must either never die or it must produce enough offspring to replace those that are lost. For a species to grow it must naturally produce more offspring than are lost.

So the feedback loop and pattern must repeat itself. It doesn't matter what the pattern is, just that it must exist. If the loop stops then another species may thrive in its place but if nothing does then death could become permanent. I would say that life is the presence of a pattern, death is the absence of one. For example many organisms might survive but not produce viable offspring. So the life was fleeting but the death far more permanent.

The question of why all organisms need to reproduce and then die instead of living infinite lives seems simple for me as a programmer. Any application I create is only perfect for the fleeting time that it meets the needs of the environment. It's guaranteed that within just a few years the very foundation on which it is built will be out of date and require reworking. You can maintain and fix the application for many years and get good service but sooner or later the cost of reworking it becomes greater than building a new one from scratch. I generally rebuild every few years for this very reason. hence I kill my old application and framework that my new one shall live. Each new version competes better than the last because I've learnt more and the environment it's built from and sits in can do so much more.

So I believe nature has the same problem and came up with the same solution millions of years before we did for essentially the same reasons. Every organism is built with the best technology of its day. Due to constant change this technology will eventually become out of date. Thus a pattern which ensures the species is built using current and different technologies is a pattern that ensures survival of the species at the expense of the individual.

So how did it all start?
Well, it's through learning how stars created the elements that I figured out my idea for how the early patterns of life emerged. Again, the process of tars creating elements is based on energy exchange, in this case entropy. The disorder that entropy creates matches the constant change we see today. This leads to energy, at the scale of a star this is sufficient energy to change hydrogen to helium and then helium to a higher element. This process went on for billions of years until all the elements we now know of were formed. Longer still and the planets formed from the the gas clouds the stars produced.

Each step is part of multiple patterns that lead to it, fueled by energy and change. The stars are part of a feedback loop. The gases in space are drawn to each other to such a point that their combined gravity and friction ignites a furnace so hot it creates nuclear fusion. Fusion actually releases more energy than is used to create it and thus it becomes a self sustaining process, only dependent on sufficient elements to fuse together. When these run out the furnace dies and the stars dies in a massive explosion. Hence a star has a life cycle of birth, life and death and it eats elements and excretes other elements. So, while it's basically a massive chemical reaction, it's also like us in very basic ways. It ingests food/fuel and excretes waste product.

So that's the feedback loop that created the elements, but where did life come from?
The next step is that the elements have a habit of bunching together. We already know that gas clouds form. Well, larger elements also have a habit of joining together. I don't know all the details but you'd essentially need to add energy to the elements to get them moving and joining together. Over long enough periods you'd get elements joining together that could create basic structures that could create more of themselves. All based on the exchange of energy.

I'm a bit vague on this but the basic idea would be that a very simple ecosystem could exist where some elements would join together. This would then release energy and waste products, protons and other things. This would encourage other elements to join together given this new energy and waste. Basically chemical reactions would occur. The pattern being that one reaction triggers another, which then triggers another until a loop occurs. If no loop occurs the pattern dies. But eventually a pattern or several patterns must have occurred that, like the furnace in the star, just found life and wouldn't stop. Again it's a very primitive life but it's a chain of chemical reactions looped together that balance each other and become self sustaining. 

I believe this would have happened away from earth, most likely inside a meteor. There it would be sufficiently protected from the elements of space but still bombarded with radiation. The radiation providing the energy to fuel the chemical reactions taking place inside. Over billions of years the life inside would have evolved dramatically. The original elements and chemical reactions being replaced by the waste products and new feedback loops(chemical reactions) evolved in their place. Newer more complex molecules and combinations of molecules could be produced and made available.

Now the question is, what were these elements?
At first I felt it should be viruses, because I feel they developed before cells were evolved to protect the DNA in the nucleus. Before that I wondered whether DNA or RNA was actually its own lifeform and viruses were simply forms of life that chose not to form their own protective cell wall. But now I'm not sure that DNA was first. Maybe life forms began but kept dying because they had no efficient mechanism for reproducing. DNA and RNA may have come along much later as an efficient means of storing the information to create life. I say this because I wonder where the cell wall came from along with mitochondria, organelles and everything else.

Where did it all come from? However I've just realised it could be its own answer.
All these chemical reactions could have occurred in a liquid inside a meteor. The cell organs could all have developed separately at first through basic chemical reactions as previously explained. They then benefit from each others presence and at some point find even greater strength by being encased in a cell wall and using DNA/RNA to store their data and replicate themselves.

All of this could occur in a meteor or on earth, that's beyond what I know. The point is simply that the constant supply of energy to fuel the chemical processes is what lead to all this. The constant change ensured processes that didn't end in some kind of loop died out. Those that did, flourished and encouraged further evolution of the elements available for life and the processes by which energy exchange could be created. So it became a self fulfilling cycle.

The original forms of life either adapted or died due to the new environment they created. That seems to me to be the essential fact of life. It's not always the species causing its own extinction, a meteor crashing into a planet breaking open its core would change the gases and elements available to the life forms and the energy that they're exposed to. Some life forms would die, others would manage to adapt and survive. In my view one or more of these meteors crashed into earth and delivered life forms with chemical reactions that ultimately changed the envrionment and atmosphere of this planet into what it is now.

We may find that these original life forms have all died out, or they may exist among us. This may even be where the cell wall became important. As a basic protection from the outside world. When I watch videos of life inside a cell it very much feels like a walled city with all inhabitants thriving on the inside, well protected from the dangers outside. Each organelle within the cell seems just like an inhabitant of a city with their own role to play, each contributing to the overall balance and productivity.

Anywho, I feel I've explained the basis of how I feel life may have begun. It may have taken a different route but I feel that the process involves energy sustaining repeated chemical processes. As life evolves it begins to actively search out energy which leads to predator and prey and food chains. Balance is constantly being sought because new life can be created on the waste of others but any balance that is achieved can be upset by any number of events. Death occurs when the patterns of life that produce energy exchange stop. We see this when the sun is blocked by a meteor crashing to earth and creating a huge dust cloud, or when all the food an organism uses for fuel gets used up.

It explains for me why crocodiles have hardly changed since dinosaur times. Essentially because they're so efficient with energy they feel no pressure to change, yet mammals who are so wasteful of energy feel constant pressure to change.

So if you feel I've missed something that's glaringly obvious, or just don't agree with what I'm saying then please post a comment. I've just put down what makes sense to me. I can't claim it's correct it just seems to explain so much to me.

I also like that it's essentially very simple. It says that the origin of life is in energy. Life exists when energy is being changed to a different form in a pattern that repeats itself over time. The presence of life inevitably changes the environment around it. Often to the point where energy giving processes become difficult. This creates pressure to evolve. Either existing or new organisms will find life and convert energy from the new environment that is created. This in itself is a repeatable process. one which must have failed on all planets except our own. Unfortunately it suggests that life has begun and died all around our universe. It may still thrive elsewhere, but to our knowledge this is still the only planet where the patterns of life have yet to stop running.

edit 12th november 2010
in Horizontal and vertical: the evolution of evolution I think they're arriving at some of the conclusions I have and in my post evolution is everywhere I've added to this concept further. 

IBM Pointstream adds in depth data analysis to tennis

Nice to see IBM is pushing out new data analysis tools for tennis. They're debuting the pointstream service at the US Open 2010. I played with it a little. Boy does it have a lot of data. From a momentum meter to a representation of every single point in the match. You can easily see serves, rallies, break points, double faults and you can select a bunch of ongoing matches as well. Fascinating