[FIXED] Best way to process a huge HTTP JSON response

Issue

What is the most efficient way to parse a JSON reply from a server that is return as 1 huge line? I don’t want/need to “map” this JSON to some custom/business objects as I need only specific parts of the structure.
Mainly finding a specific tag and its values among a set of arrays.

UPDATE:

I am looking for an efficient way to parse an JSON response from an internal server.
The response is sent as a huge single line. The file is so big that trying to understand the structure using brower’s json plugin was still difficult as the browser was “stuck”.
For my needs I want specific bits of information and do not want to map the JSON to actual business object/classes as it will be too much unneeded work. What I currently do for testing this out (I have removed exception handling for clarity):

HttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
HttpResponse response = null;
response = client.execute(httpGet);
StatusLine statusLine = response.getStatusLine();
int statusCode = statusLine.getStatusCode();
StringBuilder builder = null;
if (statusCode == 200) {
          HttpEntity entity = response.getEntity();
          InputStream content = entity.getContent();
          builder = new StringBuilder();
          BufferedReader reader = new BufferedReader(new InputStreamReader(content));
          String line;
          while ((line = reader.readLine()) != null) {
                    builder.append(line);
          }
          if ( builder.length() != 0 ) {
                processJSON(builder.toString());
          }
           return builder.toString();
}

private void processJSON(String s) {
        JSONObject o = new JSONObject(s);
        String url = (String) o.get("url");

       JSONArray array = o.getJSONArray("individualroles");

       for( int i = 0; i < array.length(); i++ ) {
                JSONObject elem = (JSONObject) array.get(i);
                String fullNameForDisplay = (String) elem.get("fullName");
                String status =  (String) elem.get("status");
                String date = (String) elem.get("date");
                String fullDescription = (String) elem.get("full_description").toString();
                JSONArray contacts = (JSONArray) elem.get("contacts");
                for(int j = 0; j < contacts.length(); j ++ ) {
                    JSONObject contact = (JSONObject) contacts.get(j);
                    String contactName = contact.getString("contactName");
                    String displayName = contact.getString("displayName");
                    String realAddress = contact.getString("realAddress");

                    Log.i(“MyApp”, “contactName = “ + contactName + “ displayName = “ + displayName + “ readAddress  = “ + realAddress);
                }
         }

This literally fluds the log as the JSON response i.e. the String s has length of ~7Million characters.
Also I see in the log many statements like:

I/art﹕ Background sticky concurrent mark sweep GC freed 389(14KB) AllocSpace objects, 3(851KB) LOS objects, 0% free, 2MB/2MB, paused 30.651ms total 77.127ms

For testing this what I did is that in the fragment’s onCreate I do:

 Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                Log.i(“MyApp”, (new JSONParser().get(JSON_URL_SERVER)));
               }
        });
        thread.start();

Is there any issue e.g. with memory and GC with this approach? How can I improve this?

Solution

You should use something that parses the inputstream during reading.

Similar to SAX parsers for XML there are such implementations for JSON too. There's Jackson for one and alsp json-simple which has a quite simple approach to it, expained here.

If I understand your code right, you are mainly looking for everything under key "individualroles". So with this SAX-like parser you'd implement a ContentFinder like the KeyFinder in the example and that is invoked by the Parser as soon as the required key is reached in the inputstream. Now you can handle that part of the JSON and then you can end the parsing/reading alltogether.

I'm sorry I cannot provide a more detailed explanation but I haven't used that particular library myself, it's just knowledge from SAX-parsers mapped to JSON I can provide.

Answered By - Ridcully
Answer Checked By - Marilyn (JavaFixing Volunteer)