Python text extraction

You need to track what section you are in as you are parsing the file. There are cleaner ways to write the state machine, but as a quick and simple example, you could do something like below.

Basically, add all the lines for each section to a list for that section, then combine the lists and do whatever at the end. Note, I didn't test this, just psuedo-coding to show you the general idea.

authors = []
title = []
section = None

for line in articles:
    line = line.strip()

    # Check for start of new section, select the right list to add to
    if line.startswith("AU"):
        line = line[3:]
        section = authors
    elif line.startswith("TI"):
        line = line[3:]
        section = title
    # Other sections..
    ...

    # Add line to the current section
    if line and section is not None:
        section.append(line)

authors_str = ', '.join(authors)
title_str = ' '.join(title)
print authors_str, title_str

Initial Understanding

Based on your example, I believe that:

• The text is provided in lines.
• The example text appears to have too many newlines, possibly an artifact of it being migrated from DOS/Windows? If so, either CRLF processing is needed, or alternate lines should be ignored.
• The lines are divided into sections.
• Each section is delimited by a two-letter uppercase tag in columns 0,1 at the first line in the section, and continues until the start of a new section.
• Each line has either a tag or 2 blank spaces, followed by a blank space, in columns 0-2.
• The artificial section delimited by tag ER marks the end-of-record.
• The ER section contains no usable text.

It may also be the case that:

• Records are begun by the FN tag.
• Any text encountered outside of a FN / ER pair can be ignored.

Suggested Design

If this is true, I recommend you write a text processor using that logic:

• Read lines.
• Handle CR/LF processing; or skip alternate lines; or "don't worry the real text doesn't have these line breaks"?
• Use a state machine with an unknown number of states, the initial state being ER.
• Special rule: Ignore text in the ER state until a FN line is encountered.
• General rule: when a tag is seen, end the previous state and begin a new state named after the seen tag. Any accumulated text is added to the record.
• If no tag is seen, accumulate text in the previous tag.
• Special rule: when the ER state is entered, add the accumulated record to the list of accumulated records.

At the end of this process, you will have a list of records, having various accumulated tags. You may then process the tags in various ways.

Something like this:

from warnings import warn

Debug = True

def read_lines_from(file):
    """Read and split lines from file. This is a separate function, instead
       of just using file.readlines(), in case extra work is needed like
       dos-to-unix conversion inside a unix environment.
    """
    with open(file) as f:
        text = f.read()
        lines = text.split('\n')

    return lines

def parse_file(file):
    """Parse file in format given by 
        https://stackoverflow.com/questions/54520331
    """
    lines = read_lines_from(file)
    state = 'ER'
    records = []
    current = None

    for line_no, line in enumerate(lines):
        tag, rest = line[:2], line[3:]

        if Debug:
            print(F"State: {state}, Tag: {tag}, Rest: {rest}")

        # Skip empty lines
        if tag == '':
            if Debug:
                print(F"Skip empty line at {line_no}")
            continue

        if tag == '  ':
            # Append text, except in ER state.
            if state != 'ER':
                if Debug:
                    print(F"Append text to {state}: {rest}")
                current[state].append(rest)
            continue

        # Found a tag. Process it.

        if tag == 'ER':
            if Debug:
                print("Tag 'ER'. Completed record:")
                print(current)

            records.append(current)
            current = None
            state = tag
            continue

        if tag == 'FN':
            if state != 'ER':
                warn(F"Found 'FN' tag without previous 'ER' at line {line_no}")
                if len(current.keys()):
                    warn(F"Previous record (FN:{current['FN']}) discarded.")

            if Debug:
                print("Tag 'FN'. Create empty record.")

            current = {}

        # All tags except ER get this:
        if Debug:
            print(F"Tag '{tag}'. Create list with rest: {rest}")

        current[tag] = [rest]
        state = tag

    return records

if __name__ == '__main__':
    records = parse_file('input.txt')
    print('Records =', records)