I went to the #EsriUC and All I Learned Was Not to Trust Esri Products. Again.

Okay, I know. Old news, right? Those of us who have been using Esri Software to build real-world solutions to managing spatial data know there is always SOMETHING that requires a work-around. I really like the idea of what Esri does, but I must spend 10-15% of my time trying to implement products and features that should have NEVER been released. Not because the idea behind them is bad, but because the execution is flawed. And, the worst thing? They know they are doing it!

Case in point:

For the last 8 years, I have been trying to manage a rolling street  tree survey of the some 30k street trees in The City of New Haven using various Esri Products. I use this system as an introduction to field data collection for the 150 incoming Forestry Students we get in the Graduate program, every year. But more often than not, what they learn is how frustrating, finicky and fruitless these platforms can be.

First, ArcPad.

Again, the idea of ArcPad is great. And, it works just fine in the UC Demos when they are using a dataset with 30 points, rather than 30k. But, in the real world, we don’t have datasets with 30 features. Hell, just the bike rack dataset for the university I work for has 300+ features. SO for eight years, I’ve been breaking ArcPad. I’ve developed something of a reputation with the ArcPad Developers Team, too. Because when it doesn’t work the way THEY SAY it’s supposed to work, I call bullshit and I call it loud. I’ve struggled the entire time to implement a system that provided us what any real-world infrastructure inventory system needs: Disconnected Editing, Relational Data, Flexibility, etc…

We’ve been through lots with this platform, too. Finally, at about version 8, when the ArcPad Server Extension was released, we sort of got a solution. And, as ugly as it turned out to be, we actually got a way to edit relation databases in the field. But Esri can’t leave well enough alone, and discontinued the ArcPad Server extension, BEFORE their plans to integrate feature service sync were even ready to release. Finally, at 10.2, they release Feature Service Sync in ArcPad. And, as always, I spent months trying to get it to work. Dozens of hours on Esri Support chatting and speaking with tier 1 wastes of my time, and getting nowhere. Finally, the EsriUC is here, and I get to sit down with some of the ArcPad Team to troubleshoot this thing.

First, I’m told

“Oh, you can’t use it on 30K records.”

“Well why the hell not and where does it say that? It’s not a hardware issue, anymore, so what is your excuse?”

At any rate, we arrive at a workaround to sideload the data into ArcPad on a machine and use the “Incremental Download” Feature, documented here, so ArcPad won’t be overwhelmed with my real data. Testing in front of the ArcPad Team Member, she was at a loss about why it wouldn’t work. We investigate my platform, everything is at 10.2.2 (Except ArcPad, which is always at least a version behind), etc… and should be working. SO, I escalate the issue by walking over to the Esri Mobile Team Tech Lead and ask about it.

“That feature isn’t imlemented, yet”

“Wait, WTF are you talking about, It’s documented right here?” (I show her on my laptop).

“Oh, that shouldn’t be there. That doesn’t work.”

At this point, I’m seething, but I’m keeping it together, because, well, I don’t know why since they haven’t ever earned it.

Fine, so my next visit is to the Geodatabase Team to see how I can workaround the fact that I can’t do disconnected editing in Collector for ArcGIS with versioned data.

“Do you need versioned data?”

“Well, I do to use archiving, so I can see what my trees were, and not just what they ARE.”

“No, we discontinued that at sub-version blah blah.” (This is how you have to learn about these things that change the way you do things, since there is no way for one human to be comprehensively knowledgeable about the entire platform. Believe me, I try).

“Beautiful! That solves my problem! Thanks”

So, I can just ditch ArcPad altogether now, and despite making about $5000 worth of Windows Tablets useless, moving forward, I have a solution.

Except. It’s Esri and it’s never that simple. That’s when I got the email from another tree survey project I consulted on, recently.

I spent the afternoon working with a group of Nature Conservancy interns last week. I was really stoked about the new offline editing capabilities in the Collector for ArcGIS, and the ease of setup when paired with an ArcGIS Online for Orgs account! From brainstorming the schema, to creating the geodatabase to pushing it to AGO and customizing the data collection forms in our WebMap, to testing it outside and syncing to the Hosted Feature Service, it took us about 3.5 hours. Some of you know how amazing that is.

But then, The Collector for ArcGIS Team pushed an update and everything went haywire. The app did nothing but crash, and five people sat on their hands for a week because their survey system simply would not work.

I suggested uninstalling, reinstalling and redownloading the app and data, which they did, numerous times. None of it worked. Finally, I get an email with the Subject: Tablet Horrors. Basically, they are looking at Fulcrumapp, the system they looked at initially, and which I suggested wouldn’t integrate as well with their existing workflows (they, like so many large Orgs are an Esri shop). I responded, “I’m going to check with the Devs, here, and see what is up. What they suggest.”

I made the last “Collector for ArcGIS Introduction” session at the EsriUC and waylaid the Devs to ask their advice.  I explained the issues and they stopped me halfway through…

“You’re using a new Nexus 7, aren’t you?”

“Yeah, of course. It’s the best Android Device out there. Why?”

“Yeah, last weeks update introduced a bug that broke Syncing on WiFi on the Nexus 7. We have a fix, but with the UC we’ll be too busy to release it until next week.”

“Can I get a copy of the APK? I’ve got a crew of 5 interns sitting on their hands for the last week because of this, learning nothing but not to trust your software.”

They looked at me like I had two heads and basically repeated that it would be out next week. So, in a nice way, “fuck off.”

Now, I used to run my own business, and if my people had knowingly done something that effected my customers, nobody would have been going to any fucking parties until it was made right. Further, when I spec a system based upon the FLAGSHIP DEVICE OF THE ANDROID OPERATING SYSTEM, I should be confident that it has AT LEAST been tested on those devices. I do a pretty damned good job of making myself look like an asshole, I don’t need Esri’s help.

So, that’s two survey solutions, stopped dead, within 2 hours of one another. Why? Because you can’t trust Esri software. I knew this going in, though. Hell, I’ve developed a repetitive stress injury from crossing my fingers for years when using Esri software.

I forwarded the series of frantic emails between myself and the Nature Conservancy to the Esri Mobile Team Tech Lead with the comment “#JESUSFUCKINGCHRISTWILLITEVEREND?”

I wish I could invoice Esri for all the time I’ve wasted on this shit.

I’m still waiting for a response.

And, that Collector update too.


Learning to Walk, Again.

So, lately, I’ve been realizing just how crippling a dependence upon the expediency of using the Esri suite of GIS software really is.  In my position as a GIS Specialist and Instruction coordinator at an Ivy League school, it’s been easy to let myself just design every solution based upon the availability of some “out-of-the-box” solution from Esri.  I put “Out-of-the-Box” in quotes here, for a reason.  For years now I’ve felt like punching every Esri employee I hear say that phrase during a demonstration.  It’s nothing personal, it’s just that the phrase is disingenuous, at best.  Unless you are willing to compromise on the requirements of a particular project, whether it is field data collection, data distribution, analysis, metadata creation… whatever, there is really no such thing as an “Out-of-the-Box” solution, because there is no such thing as an “Out-of-the-Box” problem.

Ok, if all you need to do is put your vacation photos in Google Earth, or something trivial like that, there are a few “Out-of-the-Box” solutions.  But the projects that come across my desk are rarely that straightforward.

Take, for example, a project that has been making my life miserable, periodically, for a few years.  It would seem fairly straightforward to survey street trees in a city, right?  But I’ve been struggling to create a decent solution for just such a survey, since 2007. The trouble has been that there has never been a single “Out-of-the-Box” solution to all of the requirements of the project, so that the last 6 years have been one compromise after another to meet the needs.  In a nutshell, the project has these requirements:

  1. Multi-user editability (10-20 users)
  2. Disconnected (our University won’t pay for cellular service) editing.
  3. Cached custom basemaps
  4. Preserving historical records of each tree.
  5. Immediate access to data in various client applications (web, desktop, etc…)

Now, call me crazy, but you would think those wouldn’t be all that difficult to build a solution for, right?  Well, try doing it with Esri’s “Out-of-the-Box” solutions.  The obvious solution to the need to preserve historical data was to create a relational database, with LOCATIONS separated from the data on the state of the location (presence of a live tree, dead tree, stump, species, DBH, etc…) in a separate, but related INSPECTION table.  Obvious, right?  Well, I’ve been listening to Esri Dev’s promise and fail to deliver a working solution to related table editing in the field for all of the 6 years I’ve been working on the project.  Even the one solution (ArcPad) that PROMISED this kind of functionality never worked properly and simply choked under the strain of our 36k locations and 75k+ Inspection records, plus a custom basemap.  Last summer at the ESRIUC 2012, I asked when the much touted ArcGIS Online for Orgs and it’s mobile applications for iPhone/iPad/Android would be supporting related table editing, and was told, “Oh, we’re working on that and it should be out by October.” The crickets are still chirping on that one.

I’m still at a loss as to WHY Esri still hasn’t addressed this GAPING HOLE in all of the supposed “Field Data Collection Solutions.”  Have these people ever collected data in the field?  Do they really NOT see the value of a TRULY RELATIONAL DATABASE MODEL?  The only answer I can come up with is “NO” given the anecdotal evidence above, as well as their piss-poor implementation of SQL and relational database mechanics in ArcGIS platforms of all flavors (Desktop/Server/Mobile, etc…).

And, the Esri solutions have just become so overbloated and unnecessarily complex that it’s nearly impossible to make real use of them.  I co-taught a directed study with Dr. Dana Tomlin this semester for a student who was interested in learning more about deploying the type of Enterprise GIS and Data Collection solutions that ArcGIS Server and the new ArcGIS Online for Organizations have been promising.  Now, I’ve implemented production SQL Server>SDE>ArcGIS Server tech stacks about a dozen times  over the last 6 years for various projects, but for the first 3 weeks of this poor kid’s directed study, we were completely lost in the new SDE installation procedures, etc… Thank Imhof for the ESRI Forums, where the masses go to wonder what the f*$k is keeping their “Out-of-the-Box” solutions from working correctly, and try to help each other work things out.  Buried about 20 replies into an obscure thread we finally found the answer… turns out THE ARCGIS SERVER STACK DOESN’T NEED SDE ANY LONGER!  Now, at the end of the semester, we finally have a working prototype of the system, with exactly ONE feature class and feature service AND A DAMNED 40 PAGE DOCUMENT EXPLAINING THE TECH STACK!  That, is ridiculous.

So that brings me to the title of this post:  “Learning to Walk, Again.”  I started using ArcINFO as a command-line Unix program, back in 1996 at the University of North Texas.  In 1997, I took my first workshop to learn how the new GUI-based ArcView worked.  I wish I’d never left the Command Prompt.  Really, the GUI is just a crutch.  Does it make things faster? Probably not.  Does it make it easier to learn the software? Maybe.  What it DOES do, though, is move you further and further from what makes computers so powerful for solving problems, and that is knowing how to manipulate code and creating solutions for problems rather than fitting the problems to available solutions.  So, I am embarking upon a rigorous schedule of mental therapy, this summer.  One that I hope will “get me back on my feet” again and allow me to shed what has become, more and more, a crippling addiction to Esri’s proprietary GUI-based “Out-of-the-Box” solutions.  Unfortunately, I am only now realizing that there is no such thing as an “Out-of-the-Box” problem.

Look for more posts in the coming days and weeks, about the journey.  Tag along, if you like.  I’m going to start at the beginning and work my way back up.

Baby Steps.