Steve’s Little world

Little birdies have been telling me that ArcWeb Services is no more - folded into other services and the staff dispersed. There is no more purchase options on their page - only renew. Well, Nature Valley and all you other Arcweb Service customers, I could always recommend another service that is fast, stable, and scalable. You can start for free and then move on to exactly the same API when you are ready to make some bling. Same API whether we host it or if you bring it behind the firewall; novel concept - ain’t it.  If you are interested go ahead and contact me or sales@decarta.com

This post is going to talk about using Frameworks (JSF, ASP.NET, Django, Ruby on Rails…) to build web applications and perhaps they have outlived their usefulness in many situations.
I have started to put together a demo server here at deCarta and as part of that I am going to need to write some demos. I know that when I write some MS based demos I will be using ASP.NET since that is was one of the benefits/drawback of MS development - one “blessed” web framework.

It has been a while since I have written some Java web applications and so I thought I would peruse the landscape. I have worked in JSF before but the XML hell that is the configuration files really harshes my mellow. That being said Seam seems to be an interesting framework but there is certainly a steep learning curve and I am still unclear if it runs on Tomcat. I then find Wicket which has no XML config files and everything is done through Java code - which appeals to me. I use an IDE so it can help me avoid doing the mundane and repetitive tasks which are prone to error. Most IDEs have no way to confirm that you don’t have typos in your XML or your XML declares a non-existent class. So Wicket, using nothing but Java code, feels like a step forward… until I look at the amount of code and classes needed just to do a simple form. Feeling undeterred I try to get going and then realize that Wicket documents relies heavily on Maven, and while I have nothing inherently against Maven it is yet another thing I have to learn that I am not in the mood to do now.

Now feeling a bit frustrated I go speak to Geoff and Brent, our web services and JavaScript gurus respectively, to seek words of advice. Turns out they have used Wicket and a bunch of other frameworks and they say, as somewhat devil’s advocates, that frameworks are a waste of time. They now believe that most web apps should be written as Servlets or web services which feed appropriate data to your JavaScript library of choice. I push back a bit and they do concede that for large team projects or repeatable projects a framework makes sense but they also point out that that you spend an inordinate amount of time wrestling with the framework rather than writing code.

And this is the question for the day - has the evolution of web development moved us away from frameworks as the one true way for web developers. All the web languages out there now have their own frameworks, from perl to Ruby to .NET to Java, are they overkill for most projects. Is there where REST and Ajax (and Comet soon enough) have brought us?

When I discovered Struts it was like a ray of light cutting through all the one-off web code I had written. But are we now at a point where the frameworks and the support infrastructure are just too much hassle without enough payoff. I understand sessions are another tricky thing that frameworks help handle but perhaps that is all the frameworks should do. Programming servlets to accept GETs or PUTs and having them spit out XML or JSON is really an order magnitude (WAG ) easier than taking on a framework. It seems some in the Java community are also discussing this issue (look in the middle of the Seam discussion for the grey box which is Matt’s commentary ).

So lazy web what do you think - has the time come to give up on these heavy frameworks and move to something more lightweight - REST services with an AJAX library fronting it? Or is this just a lazy person trying to get out of doing the hard work climbing a learning curve.

For those in the geo community this would like moving from an ESRI ADF or MapGuide to something like OpenLayers + GeoServer’s KML API.

I would really love to get some more data points from the community…

As of July 23rd I am no longer gainfully employed by Jones & Stokes. It was a long process that led to this event. The most important driving factor in this chain of events was that my families’ allergies were terrible in the Sacramento valley. Moving from this area couldn’t happen through Jones & Stokes and so I had to look outside.

First question from some people is going to be “How could you leave ecology after all the glowing posts you gave”. I loved the chances I got to do field work and to talk conservation biology shop with my colleagues. The problem was that I was too expensive for my level of expertise in NorCal flora and fauna. While I am good at field work and know a few things about deserts and birds - most of my ecological experience is as a landscape ecologist. My billable rate was too high to really make it feasible for me to get out in the field that often. I would suppose this is actually close to academia, where the higher up the food chain you go the less time you get to spend outside. So in the end I didn’t get to do that much field work.

Some of you may want to know if I left because I didn’t like Jones & Stokes. This is an easy question for me - I can say without hesitation say that I think J&S is one of the best natural resources consulting firms out there. I really liked almost everyone I worked with. They generally treat their employees with respect. The work is, on balance, interesting and ecology in the real world. When I started looking outside J&S I did not even think about working for a different consulting firm.

I think once I realized I couldn’t stay with J&S I started to think what I did and did not like about being in the consulting world. The biggest problem for me was the whole billable hours thang. While my goal was not that high there were several consequences of the system I did not like.

The first problem was that no matter how many hours you worked if you didn’t meet your billable goals it doesn’t matter. So if I was working 60 hours a week trying to roll out a new service or investigating something that had to be done to take our GIS practice to another level, it wouldn’t count. I mean it would count in that my supervisor would note the extra effort, but it was all secondary to making your billable hours goal.

Second, to insure that I had enough billable hours I had to take on more than I could do, thereby insuring that I would have enough hours if a project got halted or changed direction. This was an uncomfortable position for me to be in and I am not sure I could have grown used to it over time. I don’t think any of these things were particular to Jones & Stokes but are more a fact of the consulting business.

The next logical question in the series is - what am I doing now and for that I will tell the name of the company and my position title. I am now the technology evangelist for deCarta in sunny San Jose, CA. There will be a posting in the near future talking about why I chose deCarta, what I will be doing, and my goals. On of the plus side (or the downside for you if you don’t like my writing) is that I will be blogging now as part of my “official” duties. I am really psyched about my new position and I want my next post do it justice. Then again - putting the weight of doing justice on a post is a sure fire way to make it months before it gets written.

I would write more but I am finding it hard to find time to get the thoughts together. We still don’t have a place to live in San Jose or Oakland/Berkley. This lack of a very basic foundation in life - along with trying to get up to speed in a new position, is consuming most of my mental energy. If you know of a nice 3+ bdrm, 2+ bath house that takes dogs in either the willow glen/cambrian area in San Jose or in a nice safe neigborhood in Oakland or Berkely please let me know.

If you haven’t seen it go do a google route RIGHT NOW! After the route is finished start dragging points in the middle and watch the route recalculate right before your very eyes. Here is a sample if you don’t want to type places in. This has to be one of the best AJAX things I have ever seen.

You wanna start taking bets now on how long until it does traveling salesman type optimizations or tying directly into the traffic feeds or tie ins to google news?

If I were a small or medium business that needed maps and directions with some custom points of interest - I would be knocking in Googles door for an intranet license. How much do you think it would costs to set up this kind of infrastructure for myself and get it to work some time in this century.  Think about the value in this.

Oh yeah, and one more point for all us developer types , the bar has been raised yet again for what users will come to expect as “standard”. I can see the discussion now - “When I move break point in the electrical line around I want to see the power reroute in real time.”

I don’t think I even have to mention what this means for ArcGIS Server w/ network analyst or even on desktop ArcGIS…

Hat tip to Brian Flood for pointing it out. As he said (paraphrasing) - if they can do with web service calls what can they not do?

Sitting in a introduction to the OpenSource stack for WhereCamp it was kinda sad how dismissive this crowd can be of all that has come before. I think it is because most of them have not come from GIS they might feel like they have introduced all this new and great stuff. And while I agree they have definetly pushing consumer based mapping light years beyond where it was. Their ability to do this has been made possible through the narrowing of the problem space. Their is still a lot of mapping and “where” that happens outside of this space. I do think the traditional GIS crowd has somewhat brought this on themselves by portraying themselves as the “high priesthood” of GIS and if you don’t do follow along our path and do your time in the Monastarey then you are not worthy.

There could be a lot of synergy here but instead the mistrust and dismissive attitudes on both side really hurts the realm of the possible.
So there is my assessment of the attitude I mentioned in one of my earlier posts…

Why ESRI is not really talked about:

There a lot of startups - programmers without the background in academic GIS - which is the typical path to ESRI

They are really into LBS - immersive stuff - and ESRI just doesn’t really focus on this stuff. Sure they have a few pages on this but they are not really pushing into this space.

They are almost all web based and as we all know ESRI is having some problems figuring out their strategy in this space.

ESRI is a rather large infrastructure to get up and running to play in the space. Much easier to put XY in a database with some attributes and start playing. Getting the full ESRI stack in place is $ and time intesive - not conducive to playing and rapid developement. Try and rent a cheap hosting server where you can install and get running with ESRI stack quickly.

Hence - I think these kind of conferences will probably not have much ESRI presence - there is little overlap between the audiences…..

For all you ArcSDE users out there - a good reason to SP2 for 9.2 users and users of older versions should see the notes at the end…

SecuriTeam™ - ESRI ArcSDE Buffer Overflow Vulnerability

Following up on James’s post about my IM ping - it is a good time to be a Java developer if you deal with spatial problems (for other problems as well but I am just going to talk about spatial today.

There is the ESRI Stack. So if you want to go down that route with the full support and need to use ESRI technologies you are good to go.

Here is the stack that is available to you today for free (as in beer and speech) with some brief personal experiences tacked on…

IMS server: geoserver - super easy to set up, performs well, does WFS, WMS, and KML, and reads a bunch of formats

OpenLayers: not specifically Java but works perfectly fine with it and some of their demos hit against geoserver.

Datastore: PostGIS with PostgreSQL - not much experience using but a large user community, a plugin to use with ArcGIS (oh if I didn’t have billable goals), an interface to geoserver, uDig, and geotools, and a JDBC driver that can talk spatial talk

Desktop : uDig, Jump, OpenJump, and a couple more that I can’t keep track of. They are not at the level of ArcGIS but they can do quite a bit of stuff and are open so you can extend the code however you like.

Virtual Globe: WorldWind Java - easy cheezy to get running, runs really fast, and in very active development with a friendly user community.

Toolkit: Geotools and OpenMap- spatial toolkits that have been around for at least 6 to 7 years maybe longer.

IDEs: Eclipse and Netbeans - both excellent IDEs with large user communities and active corporate support.

So there you have it, a huge stack of Java spatial applications and libraries - running on all the major OS’s - running server side and desktop side.

Do you know what your cost as a consultant or developer is to entry into this space? $0 in actual startup costs. Buy yourself a laptop, leech off your neighbors wifi connection (not that I would condone such behavior), and dig in.

Java is taught in most of the CS and IS programmings as the programming language of choice so it is also easish to find developers or learn yourself. You would be in good company with firms such as Fedex, NASA, and Motorola to name a few.

Open source and proprietary solutions all with the same language. I just think it is an interesting proposition to those who want to start developing and building spatial solutions.

I am going to write a little post to toot my own horn and discuss some of the REST ideas people, especially Sean , have brought up. About 2 years ago I started experimenting with REST shortly after starting at ESRI. It was prompted mostly by blog posts by John Udell and the advent of GreaseMonkey. You can see my original post here (we gotta use the Wayback Machine because I no longer host my own blog). Some of the links still work but the A9 OpenSeach no longer works because I don’t have a Tomcat instance running outside a firewall. If you go to the full Wayback archive for my blog that includes that date, please scan down to “feeling a bit slow on the uptake” and the post right on top. I think the whole Google removing their SOAP api is an example of providers seeing the problem of people seeing your data without getting their eyeballs to your property. And that whole recombination thing is the point of Yahoo! pipes, which is why I thought it was so great when it came out.

What I had done internally at the time was write a greasemonkey script that scraped an Amazon book page for an ISBN number. The script then did a REST request to a controlling servelet on my server, which then made a REST request to an Amazon Web service for a list of resellers, then from that list requested addresses for each reseller, then took the cities and states for each reseller. Now I wrote a servelet in front of an ArcGIS Server geocoding service and another in front of a mapping service. The geocoding servelet would take a comma seperated url of cities and states and return a XML document with X and Y locations. The mapping servelet would take a comma seperated list of X and Ys in the URL and return an URL of a map of the U.S. with the dots for each reseller’s location. For the last stage this URL would be returned to the greasemonkey script that would then insert that url into an IMG tag and render the map on the page.

I can not tell you how much fun it was working on that project. The hardest part was getting the Greasemonkey script to reliably scrape the ISBN number from the page and then insert the image in the right location. I showed this around a bit, but it only worked inside the ESRI firewall (oh to have an external facing server like they do now). Working on that got me thinking about loosely coupling with HTTP requests and XML (JSON was still in it’s infancy then and I felt very comfortable tooling around with XML). Of course the code was erased when I did one of my many clean sweeps to reinstall development builds - so you will just have to take my word for it (or not).

So the ideas have been there, I just think like with most technology, GIS people are usually not on the leading edge. They need the idea to bake into other services with other examples and then build on top of that. Which I think supports the idea that GIS and mapping are usually part of a larger question, and it is only once you can get the other data in the larger question, can you then bring GIS to bear.

I am glad ESRI is thinking about putting a REST API in front of server but it is also not that hard to build one yourself. I mean we already have the SOAP API so why not wrap that in some XML parsing, web programming goodness and make your own API. One of the the beauties of REST is that it is all urls and xml (or json) and there are so many fewer interop problems than you can get with web services. But a drawback is how to establish a contract as to what the URL should look like and what I am going to send back in response. A SOAP web services gives up flexibility and ease of use for more certainty in the contract of send and receive. Both SOAP and REST have their place, it just depends on what you want to do.

It should be interesting to see if ESRI groks what REST is about or if we end up with an all roads leads to ArcObjects api.

Working with some tabular data and some point feature classes in ArcMap. I want to do a one point feature to many row join but I could also settle for a definition query where I would end up with 1 feature to 1 row

Step 1 - export the Query to a table in my personal geodb and then try to join on the appropriate fields - join works (after I follow all the rules) but it doesn’t respect the Definition query I put on the table…grrrr

Step 2 - go to the support find and that this is a known issue

Step 3 - go talk to some of the GIS analysts and they reccomend a relate rather than a join. Trouble is you can’t symbolize by a relate, it just looks like it is intended for editing.

Right now I am looking at generating 1 DBF or table per definition query I would have used above and then joining. This stinks… Does anyone have better suggestions to this issue.