How to use OT3 encoding? Does it redefine existing macros?

An OT3 encoding is listed in texdoc encguide, but it was never released (not even coded, actually): it never went beyond the proposal stage.

According to the cited document “LaTeX font encodings”, it was meant to support the Washington State University IPA fonts, but it was soon supplanted by the tipa package and related fonts, which gave origin to the T3 encoding.

The fact that it's an “old” encoding doesn't mean “old look”, whatever that means.

If you want an “old look”, you can choose the font according to your taste. For instance, Old Standard or Fell Type.

\documentclass{article}
\usepackage{fontspec}

\usepackage{lipsum}

\newfontfamily{\oldstandard}{OldStandard}[
  Extension=.otf,
  UprightFont=*-Regular,
  BoldFont=*-Bold,
  ItalicFont=*-Italic,
  BoldItalicFont=*-BoldItalic,
  NFSSFamily=OST,
]
\newfontfamily{\felltype}{IMFeENrm28P}[
  Extension=.otf,
  UprightFont=*,
  BoldFont=*,
  ItalicFont=*,
  BoldItalicFont=*,
  NFSSFamily=IMFT,
]


\begin{document}

\oldstandard\renewcommand{\familydefault}{OST}
\section{Old Standard}

\lipsum[42]

\felltype\renewcommand{\familydefault}{IMFT}
\section{Fell Type}

\lipsum[42]

\end{document}

There are also a range of choices for older font styles for 8bit engines e.g. pdfLaTeX.

Here's one example:

\documentclass{article}
\usepackage{venturisold}
\usepackage{kantlipsum}
\pagestyle{empty}
\begin{document}
\section{Regular}
\kant[1]
\section{Bold}
\bfseries
\kant[1]
\section{Variants}
\normalfont\textti{Small-Caps Titling}
\mdseries
\itshape
Italics
\bfseries
Bold Italics


\noindent
\normalfont
Asterisked Asterisked Asterisked
\bfseries
Asterisked Asterisked Asterisked

\noindent
\normalfont
\altstyle
Asterisked As+teris+ked As*teris*ked*
\bfseries
Asterisked As+teris+ked As*teris*ked*

\section{Figures}
\normalfont
0123456789 \textit{0123456789}
\bfseries
0123456789 \textit{0123456789}
\textti{0123456789}
\end{document}

Using fancy dropped capitals at the start of chapters etc. can also contribute to the overall feel of the text's style. For example,

\documentclass{article}
\usepackage{romande}
\usepackage{lettrine}
\usepackage{GotIn}
\pagestyle{empty}
\begin{document}
\lettrine[refstring]{\gotin{T}}{his} is just some random text which should take more than one line so that the dropped capital has something to wrap around it and does
not just hang peculiarly in the middle of typographic nowhere.
\end{document}

A reasonable variety of styles are available from the initials and cfr-initials packages.

Ornaments can also be used to create breaks in text, for example, with a more old-fashioned feel. For packages, see, for example, pgfornament, pifont, adforn etc.

As explained in comments, you cannot have figured out how to use OT3 in LaTeX unless you wrote the encoding file yourself and configured LaTeX to use it.

\usepackage{ipa}

does not use the OT3 encoding. It rather does this:

\DeclareFontSubstitution{U}{ipa}{m}{n}
\def\ipa{\protect\@pipa}
\def\@pipa{\fontencoding{U}\fontfamily{ipa}\selectfont}

A list of macro definitions which utilise \ipa then follows to provide named access to characters in the relevant font.

U is a 'raw' font encoding, typically used for symbol fonts. It simply uses whichever encoding the symbol font happens to use, so that the first character is the glyph in the slot with index 0, the second is in the slot with index 1 and so on.

I could not immediately see any re-definitions of macros defined in the LaTeX format but, since they are all defined with \def, there is certainly danger of this. Moreover, there may well be conflicts if ipa is loaded in addition to other packages which extend the characters available. For example, it defines \corner, \eth, \open, \dz` etc., which are all quite likely names.

I do not know why the package does

\DeclareFontSubstitution{U}{ipa}{m}{n}

and I do not believe this is wise.

It would be better if the package went about things a bit more cautiously. However, it is an old package and probably such practices were more common and possibly less problematic at the time.

I would not recommend using ipa in new documents and it will not give your document any particular appearance, except insofar as the ipa font has a particular character. But it would be much better to find an opentype or truetype font with similar characteristics, if you simply like the font. Alternatively, you could easily write a few macros to provide safe and convenient access to the ipa font, if you really wish to use that specific font and no substitute satisfies your desiderata.

Even just copying ipa.sty to myipa.sty, removing the substitution line and replacing \def with \newcommand for the character macros would be much safer than using the package as-is.

At the same time, if you do not require conflicting packages, there are far worse packages you could use than this one. But, of course, you have to remember you are using it and check the other packages you are using to avoid unexpected surprises.